Big Data Engineer - Job Description
Have you ever received an email or a text offering to sell you something you did not need? 98% of this type of reach-out by companies goes to people who don’t need it. Leadgence identifies for B2B companies potential customers who need their product now, helping increase their sales conversion rate.
We do this by analyzing large amounts of data and this position is a key hire within the R&D team.
In this role, you will tackle the challenges of collecting a huge amount of data across the web, process it and analyze it. The best person will have a programming background and experience with open source projects in the big data domain.
What will you do?
- Build highly scalable data pipelines and data-sets
- Enhance our data architecture to balance scale and performance.
- Generate required information for data across the web
- Build, optimize and automate data processes to support business requirements
- Work closely with the CTO to make sure our infrastructure supports the ongoing growth of data
- Design and implement the required infrastructure for running data pipelines at scale on AWS / Google Cloud
Who are you?
- Software engineering mindset and ability to write elegant, maintainable code
- Analytical mindset to understand business needs, and come up with engineering solutions
- Experience balancing complexity and simplicity in terms of pipeline design
- Expertise building data pipelines on large complex datasets using Spark, Hadoop or other open source frameworks
- Expertise in one or more programming languages: Python (an advantage), Ruby, Java, Go
- Strong SQL skills and experience in working with various databases or query engines (Spark SQL / Dataframes, AWS Athena, Google Big Query)
- Experience with data workflows management tools: Airflow, Luigi, etc.
- Experience with AWS cloud services: EC2, ECS, EMR, Athena, S3. (or Google Cloud equivalent)
- Excellent communication skills to collaborate with cross-functional partners and independently drive projects and decisions
- Advantage - Experience in:
- Kubernetes / AWS Fargate
- Web Crawling
A few more things to know:
Our culture is unique and we live by our values. You will need to be comfortable working in the most agile of environments. Requirements will be vague. Iterations will be rapid. You will need to be nimble and take smart decisions.
To apply email us at: [email protected]