tag:blog.barac.at,2013:/posts Ahmad Baracat 2020-08-15T12:41:01Z Ahmad Baracat tag:blog.barac.at,2013:Post/1571113 2020-07-09T14:17:42Z 2020-07-11T11:54:23Z How Machine Learning Pipelines Evolve Based on your Business Maturity

Disclaimer

  • This article is intentionally high-level and devoid from technical jargon.
  • That said, at times, I will mention a few technologies/services to give you more concrete examples. For these examples, I will be referring AWS services but you should be able to find similar offerings on other cloud providers.

The audience

This article is an overview of how an ML pipeline would look like depending on the stage your team or company is in and what to keep in mind at each stage. The main audience are engineers & scientists working on ML productionization and executives or their equivalent in startups (ex: CTOs) who want to have a broad overview of the topic.

Background

During my tenure at Amazon, I was very lucky to be involved in multiple projects designing and building ML pipelines.

In 2017-18, I have designed and built an end-to-end ML pipeline with the different components discussed below for Amazon Advertising: the team responsible for running all personalized display advertising on behalf of Amazon worldwide with millions of Transactions per Second (TPS) and <100 ms latency.

In 2018-19, while working on Amazon Alexa, we had to manage the deployment of our models, but being a newly formed team, we faced different trade-offs as compared to Amazon Advertising.

During the past few years, I have also consulted startups and individuals on managing their ML engineering-related problems.

Why Invest in an ML Pipeline

Imagine you are Amazon Advertising and you are using ML models to predict how much to bid on a given Ad slot, or you are Amazon Alexa and you want to filter down offensive images on multi-modal devices, or you are a startup in Egypt building eKYC solutions and you need to have models to analyze national ID cards. In all of these cases, ML models are a core component of your value proposition and technical barrier to entry.

For these reasons, you need a reliable way of managing them. Managing in this case would not only mean building a reproducible way of deploying them to production, but also keeping an eye on these models using monitoring and alarming as well as other aspects discussed below.

What is an ML Pipeline

From a bird’s-eye view, an ML pipeline has a few components.
  1. Dataset generation: responsible for fetching, joining and aggregating data from the different sources, cleaning the data, applying different filters (ex: subsampling) and generating cleaned training and validation datasets
  2. Training: responsible for training the ML model, performing hyper parameter optimization and generating a trained model 
  3. Validation: responsible for running the trained model on the validation dataset and generating validation metrics
  4. Deployment: responsible for copying the trained model to production service (if the model passes certain validation criteria and is outperforming the current model in production)
  5. Monitoring & Alarming: there are at least 2 sets of metrics and alarms depending on the role:
    • Engineering: responsible for metrics such as model latency, memory usage, CPU usage, etc.
    • Science: responsible for metrics such as model bias, input or output distribution drift, etc.

Business Maturity/Stages

Note: It is crucial to understand and be explicit about the trade-offs involved in technical decisions. That’s why I will highlight a number of trade-offs associated with each stage

Starting Up

(doing things that don’t scale is the name of the game)

If you are just starting up, don’t be ashamed of using the least friction path for having an ML pipeline. At this stage, you are not sure yet if it is worth it to invest engineering effort building a robust ML pipeline. What you need at this stage is a cheap & quick way of getting models from development environments to production. This will allow you to assess how much value is it creating for your business. In essence, you are trading speed of experimentation with robustness.

For that reason, dataset generation, training, validation and deployment could just be a series of Python scripts that you run manually following a runbook (checked-in in your Git repo or posted on your internal wiki). Depending on your use case, monitoring & alarming may be a bonus point. Whichever path you choose, make sure that there is a reproducible and documented way of getting models into production. This is crucial and will come handy when things go awry.

Note that if you are already familiar and can move fast by using off-the-shelf services like SageMaker then by all means use it. The important thing at this stage is to optimize for moving fast and experimenting.

Short to Medium Term

Once you have validated that having an ML model in production actually provides value for your business and hopefully have a sense of how much value it is creating, you can justify investing engineering effort in making the ML pipeline more robust.

Note that, at this stage, you still value moving as fast as you possibly can and thus you want to spend the least time possible on engineering or maintenance of ML pipelines. For example, you would rather spend more engineering time building more valuable features for your customers.

For these reasons (improving robustness while keeping the investment low), you should consider using off-the-shelf services like SageMaker (which supports all core components of ML pipelines mentioned above). Such services will allow you to have robust ML pipelines with auto-scaling, ease of rollback, etc. without investing too much engineering time. Note that there there is still effort to be invested in setting things up, but for the most part, once that cost is paid, it should be “hands off the wheel” experience in terms of deployment and scaling.

However, SageMaker will cost more than running your own infrastructure on EC2 for example. The trade-off then is the cost on one hand and robustness & speed of experimentation on the other hand. That cost can be justifiable at this stage since you are saving precious engineering time.

Long-Term Investment (Custom Build)

Assuming everything goes well and your business is thriving (ex: hundreds, thousands or millions of TPS), you may start to care more about cost as well as having more control over your ML pipeline since the off-the-shelf solutions may not be enough in terms of feature support. You are also willing to invest more engineering effort to build a custom solution and to maintain it. You are essentially trading off customization, cost and adding new features with engineering & maintenance effort.

Concretely, you will borrow the features that you care about from the off-the-shelf services and add more advanced custom ones depending on your use case. For example, at this stage, the pipeline may look like follows:
  1. Dataset generation is automated and run at specified schedule (ex: daily or dynamically when input or output model distribution changes in production): a cluster (ex: Spark) is spun up, runs a script to gather, transform and output the data to a durable place (ex: S3)
  2. Once a new dataset is generated, this will trigger the training process. The output of the training step may not only be the trained model, but also other meta information (ex: hyperparameters used, dataset version, model version, etc.)
  3. Once the model is trained, a validation step is triggered and it compares how the newly trained model performance against the current model in production (ex: comparing their performance on a Golden Standard dataset but beware of dataset drift)
  4. If model passes validation, the model gets deployed to production
  5. Monitoring service then kicks in and keeps an eye on the important metrics and triggers alarms when an anomaly is detected and may support rolling back a model automatically
  6. You also have the ability to manually intervene and roll back your models

Wrap-up

I hope you got a good overview of what an ML pipeline is, why build one and a few points to keep in mind depending on the stage your company or team is in.

If you are interested in learning more about the topics discussed above or related ones, please let me know in the comments below. You can also subscribe below to get notified when I post a new article.

If you are Arabic speaking, I have created 2 short courses to get you started in Data Science as well a short course with career tips (the course I wish existed when I was in the last few years of university or just graduating), which are free to access.
]]>
Ahmad Baracat
tag:blog.barac.at,2013:Post/1571172 2020-06-06T14:12:00Z 2020-08-15T12:41:01Z My Experience Switching from Tier 2 to Tier 1 Exceptional Talent UK Visa in 2020

Logistics

Let’s get the logistics out of the way:

  • This is not an official guide, for official guidance check gov.uk and Tech Nation guide
  • This is just my personal experience so take it with a grain of salt
  • Exceptional Talent (Tier 1) visa got renamed to Global Talent visa in Feb. 2020, but they are essentially the same visa. I personally like the Exceptional Talent naming as it feels more prestigious 😇

Background Story

You can safely skip this section 🥱 if you are here for the meat 🍖

In Jan 2019, a year before I actually applied to the visa, I took a step back to think about where I want to live and what I want to do moving forward. At that point in time, I have been in the UK for 1.5 years. I have just moved to London a few months earlier to work for Amazon Alexa as a software engineer whereas previously I was working for Amazon Dynamic Advertising in Edinburgh, Scotland.

Back then I reached the conclusion that I want to work independently to answer a few questions in my mind, gain more freedom, which I really missed by working as employee and just to explore. Equally important, I needed to maximize my ability to travel back and forth between wherever I am living and Cairo to see my family. The issue was that being on a Tier 2 as well as working for Amazon constrain me from registering a business or working as self-employed or travel as I please.

I started exploring the potential of moving to different countries and cities that would allow me to work independently, but that’s not the topic of this essay. Eventually, I reached the conclusion that the UK/London was probably the best country/city, which fits my personal living preferences (warm weather, travel time to Cairo, opportunities, liveliness, among other criteria) as well as the ability to work independently if I switch to another visa.

In the beginning, I quickly discarded the Tier 1 Exceptional visa because the official requirements on the UK gov. website made me realize that I needed to be, ahem, “exceptional”, which I didn’t think I am at the time (or now to be honest 😅). Unique, yes, but exceptional, I don’t think so. This left me with only one option, which was the Startup visa. In order to get that visa, you need to be endorsed and although I tried contacting a couple of the endorsement bodies, they either didn’t reply back, haven’t started accepting applications (since it was a new visa at that time), and so on and so forth.

Since I couldn’t progress on the Startup visa, I started digging again and try to understand if I can by any chance fit the Tier 1 Exceptional visa requirements. My viewpoint shifted from “I can’t apply to this visa” to “I can potentially get endorsed” happened when I was reading the Tech Nation very detailed guide on how to apply. In there, they listed the kind of example evidences that I can present in my application. Going through these, I realized I can present many of these evidences.

Impostor Syndrome

The most important idea I want to leave you with is don’t assume that you can’t apply, validate it (i.e. read the requirements, talk to other people who applied, look at previous applications, etc.), which I believe is an important lesson in life in general. As I said, if you have a quick look at the gov.uk website, you can easily reach the conclusion that the Tier 1 visa is not for you, but if you start digging a little deeper, you may realize that you can do it.

“Don’t assume, validate” - Ahmad Baracat

Benefits of Switching

To me, the main benefits of switching from tier 2 General visa to tier 1 Exceptional Talent visa are:

  • Not needing sponsorship, which makes it easier to switch companies
  • Freedom to apply to companies and startups that don’t offer sponsorship
  • Freedom to start your own businesses in the UK or be self-employed
  • Freedom to leave your company, stay in the UK or travel outside and be able to come back

The Process

  1. Application preparation
    1. Gather evidences: If you are a software engineer, these might include open source contributions, side-projects, invitations to speak at Conferences, articles written, etc.
    2. Write your personal statement
    3. Ask for reference letters
  2. Stage 1: you are essentially uploading your prepared application to a portal (to be assessed by Tech Nation if you are applying within the digital technology field)
  3. Stage 2: you are applying for the visa and filling immigration related questions such as criminal convictions, being deported from countries, etc. as well as choosing a biometric appointment
  4. Biometric appointment
  5. Receive your new Biometric Residence Permit (BRP)

The Timeline

Even though, your timeline may vary, I wanted to share mine to give you an idea of how it might look like:

  • Few days of side-work to prepare the application
  • 28th Dec 2019: Apply to stage 1
  • 20th Jan 2020: Receive Tech Nation endorsement
  • 23rd Jan 2020: Apply to stage 2 + Visa Appointment
  • 29th Jan 2020: Stage 2 successful
  • 31st Jan 2020: Receive BRP

Fees

  • £456: Stage 1 (if you don’t get endorsed you only lose this unless you applied to both stages together)
  • £1200: Health Surcharge (I applied for 3 years 3 x £400)
  • £171: Stage 2
  • £112: Visa Appointment

If you think £2K is a lot to switch visa, let me just remind you that you are essentially buying benefits of switching, which to me justifies the investment.

Tips

  • If you decide to apply, don’t wait, requirements may change and may get harder. For example, when I was reading the guidelines back in June 2019, I only needed 2 reference letters to apply. When I was applying in Dec. 2020, I got surprised that they increased it to 3 😅.
  • When asking for reference letters, make sure you make it so easy for referee to write it:
    • Remind them of what you did and what they should write about
    • Provide them with a template that they can easily fill
  • If you don’t have the referee resume, link to their LinkedIn on the reference letter instead. For example, one of my referee, being a professor, got very busy and it was hard to get them to reply to my emails.

Application Example

If you have checked the gov.uk as well as the Tech Nation guide and you think you might benefit from looking at how a complete application might look like, check my application example.

]]>
Ahmad Baracat