Expert Systems Engineer

Apply Now

Demonware creates and provides the online services behind hugely popular video game franchises such as Call of Duty and Skylanders. We provide Matchmaking, Microtransactions, Storage and Identity & Access Management services for almost half a billion users. Demonware has offices in Dublin, Ireland; Vancouver, Canada; and Shanghai, China, and is a wholly-owned subsidiary of Activision Blizzard, Inc.

Position Summary
The Infrastructure department’s goal is to provide reliable data center and cloud infrastructure to Demonware’s Title, Product and Platform teams. We design, build and operate infrastructure in multiple data centers, including network, compute and storage with hundreds of racks and thousands of servers. We also have a significant infrastructure footprint in AWS and provide tooling for accounts, IAM, VPC and VPN access. We are also responsible for hardware inventory, DNS and all IP connectivity to our data centers including connectivity to the Internet and AWS. Some of our current projects include data center standardisation, commodity storage solutions and development of best practices for running infrastructure in AWS.

As our hosting platform is evolving we are looking for an Expert Systems Engineer to join our Systems Engineering team. As an experienced engineer, you will be instrumental in setting technical direction and best practices for our current and future needs. You will improve efficiency in our existing hardware fleet and help us deploy Infrastructure in the public cloud in a secure, scalable and repeatable way. You will help drive towards infrastructure as code using frameworks such as Ansible and Terraform to provision infrastructure in an automated way.


Responsibilities

  • Leading and actively participating in key infrastructure projects involving multiple teams
  • Troubleshooting and debugging, often in collaboration with various internal teams
  • Leading and actively participating in design and code reviews
  • Managing compute systems and storage solutions in our data centres along with other key systems such as inventory, provisioning, DNS, OpenLDAP and Vault
  • Being a key player in providing compute and storage infrastructure for virtualized and containerized workloads to other teams in Demonware, both in our physical data centers and the public cloud
  • Creating and managing tools for AWS user access management and account creation, VPC peering and cost optimization
  • Acting as an escalation point for systems and services owned by the team
  • Interact and maintain relationships with data center remote hands, third party vendors and contractors as required

  • Requirements
  • 7+ years of practical infrastructure experience working with network, compute and storage hardware
  • 7+ years of hands-on Linux administration
  • Firm understanding of compute hardware and architecture including rack
  • mounts and blades and provisioning at scale
  • Firm grasp of core operating system concepts (processes, threading, memory management, file system types, ACLs)
  • Extensive understanding of networking/distributed computing environment concepts
  • Experience in hardware qualification and performance benchmarking
  • In-depth practical knowledge of open source hardware and operating system abstraction technologies: virtualization (preferably KVM) and/or Linux containers (e.g. LXC/LXD, Docker)
  • Proven experience with architecting and provisioning services in AWS
  • OpenLDAP experience, architecting and maintaining
  • Strong scripting skills: Unix shell programming and ideally at least one of Python or Perl, to assist with identification and automation of repetitive tasks
  • Strong working knowledge of TCP/IP networking as well as common application protocols
  • Familiarity with common system services, e.g. Cobbler, PXE, DNS/BIND, DHCP/BOOTP, NFS, SNMP, IPMI
  • Proven ability to take initiative for continuous improvement
  • Excellent analytical/troubleshooting skills
  • Strong written & verbal communication skills
  • Ability to work and deliver consistently under pressure

  • Desired
  • Experience working in 24x7x365 online environments
  • Experience deploying or operating an OpenNebula cloud environment
  • Experience with configuration management systems, e.g. Ansible, Puppet, Chef, CFEngine
  • Experience working with container cluster management technologies such as Mesos and Kubernetes
  • Experience describing and operating infrastructure as code using tools such as Terraform
  • Experience enabling product teams with Continuous Integration and Continuous Delivery pipelines
  • Experience with Unix like Operating Systems apart from Linux

  • What Demonware Offers: 
  • An excellent salary, with annual performance reviews and discretionary bonuses
  • 5 weeks of vacation, standard
  • RRSP matching and contributions
  • Top-tier medical, dental, and life insurance coverage
  • Flexible working practices, based on the belief that the quality of your work is not dependent on the number of hours sitting behind a desk
  • A culture that emphasizes continual learning and improvement, with opportunities for career progression, travel, and a commitment to open source

  • Demonware by the Numbers:
  • 469+ million gamers
  • 3.2+ million concurrent online gamers
  • 100+ games
  • 300,000 requests per second at peak
  • Average query response time of <.02 second
  • 630,000+ metrics a minute
  • 132 billion+ API calls per month