Lawrence Berkeley National Lab's (LBNL) NERSC Division has an opening for a Site Reliability Engineer to join the team.
In this exciting role, you will provide a variety of engineering support services 24x7 for the primary scientific computational facility for the Office of Science in the US Department of Energy (DOE).
In this key role, you will work for the Operations Team (Ops) and ensure that NERSC resources are accessible, reliable, secure and available to our scientific users. This position also supports the Energy Science Network (ESnet) Network Operations Center during off-hours.
What You Will Do:
Using the skillset of a junior Linux system administrator (as identified in the following publication https://www.usenix.org/system/files/lisa/books/usenix_22_jobs3rd_core.pdf), and their working knowledge of the systems the Operations Technology Group has responsibility for, monitor and manage the reliability of the NERSC facility to enable continuous scientific progress of the users in three areas: computation, data storage and the data center environment. This position will work onsite for a minimum of three shifts each week and may include nights, weekends and holidays.
Using the guidelines of the book Site Reliability Engineering (O'Reilly, ISBN 978-1-491-92912-4), practice the SRE philosophy in software development and system operations.
Solve problems to ensure system services available and create automation to prevent problem recurrence with the goal of automating response to all routine service conditions.
Under the guidelines of the group's project manager, assist with developing and maintaining diagnostic tools used to support the HPC community within NERSC using programming languages like C, C++, python, java or perl. Must have knowledge of standard software development practices.
Using knowledge of the Facility Operations processes, provide input in the design of software tools, workflows and new processes that continuously improve the diagnostic capabilities of the group to ensure the high availability of the HPC services provided by NERSC.
Assist in the testing and implementation of new diagnostic tools, workflows and new capabilities for providing high availability for the systems in production. Write the documentation necessary for these new tools and train staff in their use.
Provide accurate information in the trouble ticketing system for outages, maintenance and other incidents such that the workflow and protocols can be appropriately tracked by others.
Work closely with other NERSC groups to manage maintenance, to perform tasks like upgrades, to shut down batch queues, and to manage diagnostic and notification software or generally manage a center wide outage.
Conduct periodic on call duties as necessary to support a 24x7 workflow.
Bachelor's Degree in a Computer Science or similar discipline or equivalent years of experience.
Minimum of 5 years related experience including 3 years as a system administrator or system engineering in a high-volume customer-facing environment supporting data clusters, managing the replacement of hardware, and ensuring its continuous availability to the user community. This can include assisting in the deployment of new nodes and internal switches into production, resolving ticket incidents and working with vendors on hardware warranty replacements. Hands-on experience in a Linux/UNIX or knowledge of or significant exposure to Red Hat Enterprise Linux or a Linux variant.
Minimum of three years of experience in UNIX or Linux, Networking, IT infrastructure environment and management experience in a distributed-computing environment.
Strong hands-on knowledge of the Linux shell and working in a command-line (e.g. SSH) environment.
Experience with developing tools using various programming languages such as C, C++, perl, java and Python or a scripting language with knowledge of standard software development practices.
Knowledge of the processes for standard operating procedures, and best practices for implementation and change management.
Networking: experience with network theory such as TCP/IP, UDP, ICMP (networking protocols in general), MAC addresses, IP packets, DNS, OSI layers, and load balancing.
Strong understanding of monitoring implementations and administration.
Past experience with Incident Management and a good understanding of IT service management.
Exposure to Oracle or other high end Storage Infrastructure.
Background configuring distributed, server-based or cluster-based infrastructure supporting a high volume of transactions in a Linux environment. An understanding of VM's and Containers, how to manage them and an understanding of the IoT technologies.
Knowledge of and ability to work on large data communications networks and IT infrastructure supporting highly available systems and applications.
Demonstrated ability to deliver results on time with high quality.
Strong communication skills and ability to work effectively across multiple business and technical teams.
Excellent problem solving skills with the ability to work on problems of diverse scope. Must be able to think independently, work collaboratively and contribute to an active intellectual environment. Must show good judgment and ability to schedule and lead a small group of people and/or projects. Systematic problem solving approach, coupled with a strong sense of ownership and drive.
Motivated, self-starter who can learn emerging technologies that improve data center management in areas like Jupyter, Kibana, Functions as a Service, Kubenetes, building management software, evaporative cooling and power utilization.
Experience working in a 24/7 onsite team managing large data centers or other large installations. Working off shift is a lifestyle change that should be considered by the candidate.
Experience with network security: configuring/maintaining ACLs, knowledge of firewalls
Network programming or a network certification.
A certification in a system administration area.
Be able to provide input toward creating new standards and methods for managing large-scale distributed systems.
This is a full-time career appointment, exempt (monthly paid) from overtime pay.
This position may be subject to a background check. Any convictions will be evaluated to determine if they directly relate to the responsibilities and requirements of the position. Having a conviction history will not automatically disqualify an applicant from being considered for employment.
This position supports a 24x7 operation. The work schedule may include day, swing, night shift or weekend hours depending on operational needs and a 24x7 on-call rotation.
Must be able to complete an export control permit to create an account on a DOE system.
Work will be primarily performed at Lawrence Berkeley National Lab, 1 Cyclotron Road, Berkeley, CA
Berkeley Lab is an Equal Opportunity and Affirmative Action Employer. We heartily welcome applications from women, minorities, veterans, and all who would contribute to the Lab's mission of leading scientific discovery, inclusion, and professionalism. In support of our diverse global community, all qualified applicants will be considered for employment without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, disability, age, or protected veteran status.
In the world of science, Lawrence Berkeley National Laboratory (Berkeley Lab) is synonymous with excellence. Thirteen scientists associated with Berkeley Lab have won the Nobel Prize. Fifty-seven Lab scientists are members of the National Academy of Sciences (NAS), one of the highest honors for a scientist in the United States. Thirteen of our scientists have won the National Medal of Science, our nation's highest award for lifetime achievement in fields of scientific research. Eighteen of our engineers have been elected to the National Academy of Engineering, and three of our scientists have been elected into the Institute of Medicine. In addition, Berkeley Lab has trained thousands of university science and engineering students who are advancing technological innovations across the nation and around the world. Berkeley Lab is a member of the national laboratory system supported by the U.S. Department of Energy through its Office of Science. It is managed by the University of California (UC) and is charged with conducting unclassified research across a wide range of scientific disciplines. Located on a 200-acre site in the hills above the UC Berkeley campus that offers spectacular... views of the San Francisco Bay, Berkeley Lab employs approximately 4,200 scientists, engineers, support staff and students. Its budget for 2011 is $735 million, with an additional $101 million in funding from the American Recovery and Reinvestment Act, for a total of $836 million. A recent study estimates the Laboratory's overall economic impact through direct, indirect and induced spending on the nine counties that make up the San Francisco Bay Area to be nearly $700 million annually. The Lab was also responsible for creating 5,600 jobs locally and 12,000 nationally. The overall economic impact on the national economy is estimated at $1.6 billion a year. Technologies developed at Berkeley Lab have generated billions of dollars in revenues, and thousands of jobs. Savings as a result of Berkeley Lab developments in lighting and windows, and other energy-efficient technologies, have also been in the billions of dollars. Berkeley Lab was founded in 1931 by Ernest Orlando Lawrence, a UC Berkeley physicist who won the 1939 Nobel Prize in physics for his invention of the cyclotron, a circular particle accelerator that opened the door to high-energy physics. It was Lawrence's belief that scientific research is best done through teams of individuals with different fields of expertise, working together. His teamwork concept is a Berkeley Lab legacy that continues today.