DevOps Engineer’s Guide To Jenkins Troubleshooting | 2023

As a DevOps engineer, CI/CD is an inseparable part of any given day at work and thus, being proficient with Jenkins is a must-have for DevOps in 2023. Jenkins is a self-hosted, Java-based open-source DevOps tool for software continuous integration/continuous delivery and deployment (CI/CD) used to manage pipelines that outline the server’s tasks. It supports building, deploying, and automating any project on Windows, Linux, macOS, and any programming language using packages and 1800+ community-derived Jenkins plugins. While Jenkins is fairly easy to learn, it isn’t rare to learn about errors and given its importance in day-to-day operations; you must be equally good at Jenkins troubleshooting. 

In this article, the DevOps experts at Talent500 will help you address and resolve common issues that are commonly encountered with Jenkins.

Let’s begin:

Understanding The Errors

When running Jenkins jobs, you may encounter error messages like java.lang.OutOfMemoryError in the build log. It may occur due to the following:

1. There is a leak in your Jenkins memory.
2. The existing heap space is insufficient for your Jenkins’ increasing data size.
3. Insufficient virtual memory OS kernel 
4. Need for larger headroom due to a surge in processing temporary data i.e., test reports 

Before we move forward with the solutions, let us learn how you can run an initial diagnosis when you run out of memory:

Basics Of Jenkins Troubleshooting

If you are consistently encountering this error at the same phase of the build, you simply need to increase the allocated memory. Also, if you find exit code 37 (128 + signal number for SIGKILL), your virtual memory might be inefficient, leading to termination. In this case, you may see builds terminate with exit code 137 (128 + signal number for SIGKILL). Here, you must use the dmesg command to see the log messages to know the actions undertaken by the kernel.

Exploring Approaches To Jenkins Troubleshooting 

Let us go through possible solutions for Jenkins OutOfTime errors:

The Jenkins error messages can be classified into two broad types:

1. java.lang.OutOfMemoryError: PermGen space  
2. java.lang.OutOfMemoryError: Heap space

In the first situation, you must increase the generation space utilized to hold Java object metadata. Note: If you modify the value of the -Xmx parameter, the problem will still continue to persist. This can be addressed by adding XX:MaxPermSize=512m (or further increasing the value from 512 as required). 

For instance, if the fresh extract from install of Jenkins looks like:

# arguments to pass to java

#JAVA_ARGS=”-Xmx256m”

Here’s how the extract looks after increasing the available memory to 1 Gb:

# arguments to pass to java

JAVA_ARGS=”-Xmx1024m”

Alternatively, you can install environment injector plugin and  use environment variables for each job to enable additional memory usage when it’s not utilized:

GRADLE_OPTS=”-Dorg.gradle.jvmargs=-Xms1024M -Xmx8192M -XX:PermSize=512M -XX:MaxPermSize=2048 -XX:+CMSClassUnloadingEnabled -XX:+HeapDumpOnOutOfMemoryError -Dfile.encoding=UTF-8″

JAVA_OPTS=”-XX:MaxPermSize=2048M”

However, you must not over-leverage the heap space and input too large figures as it will constrain the total memory available to the OS and other software programs. In the second scenario, you must increase the allocated memory space when the daemon starts by increasing the value of the -Xmx Java parameter to manipulate the maximum size of the memory allocation pool.

For the above solutions, you need to go to the tenth line of /etc/default/jenkins and uncomment the JAVA_ARGS setting on Ubuntu 12.04 LTS. Remember, you first need to install the Monitoring plugin to verify existing memory usage with the help of the command free -m to determine server memory size.

You need to add space between parameters when setting MaxPermSpace:

# arguments to pass to java

JAVA_ARGS=”-Xmx1048m -XX:MaxPermSize=512m”

Next, restart Jenkins using sudo /etc/init.d/jenkins restart (command line) for forced restart or using the Jenkins web interface.

The Jenkins.xml’s directory is located at /etc/sysconfig/ for jenkins-1.579-1.1 in the case of CentOS:

JENKINS_JAVA_OPTIONS=”-Djava.awt.headless=true -Xmx -XX:MaxPermSize=” 

For  openSUSE Leap 42.1 (x86_64):

/etc/sysconfig/jenkins

However, in cases where your memory maxes out when loading Jenkins or the loading time is too high, you might have a leaking memory.

Here, you might run JVM through -XX:+HeapDumpOnOutOfMemoryError to produce a heap dump. Also, you must attach the running instance. To target a particular Java process, you may use jmap -dump:live,file=/tmp/jenkins.hprof pid where pid refers to the process id of the concerned Java process.

Java profilers can also be helpful in the diagnosis process.

Reporting Bugs

To report a bug, you can run this Groovy script in Jenkins > Manage Jenkins > Script Console:

println(“Jenkins: ${Jenkins.instance.getVersion()}”)

println(“OS: ${System.getProperty(‘os.name’)} – ${System.getProperty(‘os.version’)}”)

println “—“

Jenkins.instance.pluginManager.plugins

    .collect()

    .sort { it.getShortName() }

    .each {

        plugin -> println(“${plugin.getShortName()}:${plugin.getVersion()}”)

    }

return

For more details on reporting a bug in Jenkins, click here.

Getting Help From Community

While DevOps engineers generally post heap dumps on open communities/forums, it is noteworthy that heap dumps may contain confidential information as well. In such cases, you must identify the largest retention size (often maps, arrays, or buffers) among all objects and find the path to GC root for identifying the Jenkins object that owns it.

Once you complete the above process, you may report the summary to Jenkins.io When using VisualVM, you must run jvisualvm on the server and choose the heap dump from the File » Load menu. jmap -histo. will help you identify the classes with the problematic instances or you can manually scan for them in the Classes tab. 

For instance, if you are debugging a Groovy script leak, go to the filter field, type GroovyClassLoader and double-click on the line that doesn’t have $.  Go to the Instances tab and select random instances among the group of 500s. Next, go to the References tab, right-click on this and then select Show Nearest GC Root.  Right-click the tree item you want to copy the path to and choose Copy Path From Root. For additional instances, copy and paste this text into a text file and attach it to a bug report. You may also choose to continue your inquiry into plugin source code when the former option isn’t feasible. It is noteworthy that this process is recommended only if you currently don’t have a memory profiling tool. Also, when looking for help from fellow DevOps engineers, you might need to help them with the thread dumps. 

Provided that you have the administrator permission on the system, you can obtain the thread dump by going to http://your.jenkins.server/threadDump where the agent name (for instance, abc), you must go to: http://your.jenkins.server/computer/abc/systemInfo 

However, that might not work well when Jenkins doesn’t respond to the web UI, and in such situations, you must use jstack to source the thread dump. Make sure to run the jstack through the same user profile that runs the Jenkins, and if you are required to use -F for obtaining the dump, be sure to mention it in the bug report as well.

When neither of these two approaches yields, you must send a signal to make the JVM print the thread dump.

If you have a terminal or command prompt that’s running the JVM, you can hit Ctrl+ + (Unix) or Ctrl+Break (Windows) to do this. On the other hand, if it is running in the background, hit kill -3 PID (Unix) or use a tool to send a signal (Windows). Obtain the dump from the generated log file, and you are good to go ahead.

Wrap Up

As a DevOps engineer, it is necessary to understand that reproducing the failures encountered during Jenkins test results can be a tedious process. Hence, using plugins can save you and the team a significant amount of time and effort but knowing the manual process helps you handle the situation confidently when the JVM faces a dire state.

We hope that this quick guide aids your journey as a DevOps engineer, and feel free to go through our resource-rich DevOps blog to be at the top of your game.

Join Talent500 now to find the best DevOps opportunities in 2023.