The Splunk Engineer will provide overall engineering, and administration in supporting a very large distributed clustered Splunk environment consisting of search heads, indexers, deployers, deployment servers, heavy/universal forwarders, and Splunk Enterprise Security premium app, spanning security, performance, and operational roles. This role will work closely with architect and partner support team, engineering and operations leads.
The Engineer should be proficient with recognizing and onboarding new data sources into Splunk, analyzing the data for anomalies and trends, and building dashboards highlighting the key trends. The Splunk engineer should be proficient within a Linux environment, editing and maintaining Splunk configuration files and apps. This role will work closely with architect and partner support team, engineering and operations leads. Provide analysis and implementation support for development team for new capabilities. Improve system efficiency and stability via the development of automation tools and scripts. Key Responsibilities:
- Supports troubleshooting of Splunk and ITSI application related problems
- Tune and configure Splunk
- Onboarding various application logs data into Splunk using Universal forwarders, HEC mechanism, and DBConnect
- Create custom add-ons for non-standard data sources
- Create custom alerts and reports
- Create Splunk Dashboards, Visualizations and Pivot tables for the business users
- Apply tuning recommendations
- Reporting used for troubleshooting, audit, and problem resolution
- Tuning of SPL and writing advanced SPL for Splunk searches and Visualization in Splunk enterprise.
- Participate in requirements gathering, architecting, building, and supporting operational teams
Core Technology Infrastructure Organization:
- 3 or more years Splunk 7.x/8.x Administration on Linux
- 1 or more years ITSI experience
- 3 or more years advanced shell scripting skills (python a plus)
- Solid Splunk SPL experience
- Syslog experience
- Excellent oral and written communication to be able to communicate at all organizational levels
- Self-motivated, confident, and reliable
- Willing to work in a team-oriented environment with global team members
- Must be a problem solver with root cause analysis skills who brings solutions to the team
- Ability to document installation procedures, Standard Operating Procedures (SOP), etc.
- Must be able to speak to Splunk data life cycle.
- Knowledge of privilege elevation tools
- Results oriented, business focused, and successful at interfacing across multiple organizational units at various levels
- Experience with ITSM Remedy or other ITSM tools
- B.S. degree in Computer Science, Engineering or equivalent preferred
- Knowledge of network operations and security operations management responsibilities, network Firewalls, Load balancers, and complex network designs
- Machine Learning Toolkit experience
- Knowledge of ITSI
- Experience in a high volume Splunk environment
- Streaming data solutions – Kafka
- Git and CI/CD experience
- Infrastructure as Code
- Strives to bring new thoughts and ideas to teams in order to drive innovation and unique solutions
- Excels in working among diverse viewpoints to determine the best path forward
- Experience in connecting with a diverse set of clients to understand future business needs – is a continuous learner
- Commitment to challenging the status quo and promoting positive change
- Participate in and drive collaborative efforts to advance tools, technology, and ways of working to better serve an evolving client base
- Believes in value of diversity so we can reflect, connect and meet the diverse needs of our clients and employees around the world
Leads and/or participates in the design, development, and implementation of complex system engineering activities involving cross-functional technical support, systems programming and data center capabilities. Responsible for components of highly complex engineering and/or analytical tasks and activities. Establishes input/output processes and working parameters for hardware/software compatibility, coordination of subsystems design, and integration of total system. Viewed as a technology subject matter expert; able to provide and communicate complex technology solutions across differing audiences including technical, managerial, business executives, and/or vendors.
Will have responsibility for multiple, complex projects; will direct activities of teams related to special initiatives or operations and may have direct reports. Leads the resolution process for complex problems where analysis of situations or data requires an in-depth evaluation of various factors. Exercises judgment within broadly defined practices and policies in selecting methods, techniques, and evaluation criterion for obtaining results. Information Technology degree and/or technology certifications preferred or substantial equivalent experience. Typically 7-10 years of IT experience.