Webinar replay: How carriers are leveraging large language models (LLMs) and automation to drive better decisions
Watch Now
  Everest Group IDP
             PEAK Matrix® 2022  
Indico Named as Major Contender and Star Performer in Everest Group's PEAK Matrix® for Intelligent Document Processing (IDP)
Access the Report

BLOG

The Founder’s Guide to Machine Learning: Why You Shouldn’t Build Your Own Models

May 5, 2015 | Business, Machine Learning

Back to Blog

Machine learning may be the hot new thing, but it’s still a mystery to most people. This isn’t surprising, since the field is complex and requires a high degree of expertise. In Part 1 of The Founder’s Intro to Machine Learning, we explained that machine learning involves the use of computer algorithms and mathematical models to solve real world problems, like figuring out why 34% of your users don’t return after three days of using your app. And, if you hired people with the right expertise and gave them enough time and resources to complete the job, they could build you a machine learning solution for your business problem.
So…why isn’t everyone doing it? Well, even for an expert in the field, it often requires months of experimentation to develop a model, and there is no guarantee that the model will provide a scalable solution to your business problem. That’s tens of thousands of dollars and a whole lot of risk for your business. On the other hand, hiring a third party service allows one of your software developers, with no specialised training in machine learning, to integrate the power of machine learning into your company in just a few lines of code.

In Part 2 of The Founder’s Guide to Machine Learning, we’ll compare the process of creating a machine learning model in-house with the simpler workflow of using a cloud-based service that provides pre-trained models.


The Process of Building a Machine Learning Model

There are a number of steps involved in using machine learning to solve a business problem, which are illustrated in the flowchart below. Depending on the complexity of the problem you’re trying to solve, this could take several months.

The process of building an ML model in-house

The Process of Building a Machine Learning Model

Note: Time estimates vary depending on the complexity of the problem you’re trying to solve.

After deciding on the problem you’d like to solve, the first steps of building a machine learning model involve data collection and labeling. As a rule of thumb, you and your team need to collect at least 100,000 data points (for example, tweets) to create robust models that produce good, useable results.

Then, you’ll need to label each of those data points with a value to show the model what you expect it to do. For example, in the case of sentiment analysis, you would label “1” for positive tweets, and “0” for negative tweets.

As a basic thought experiment, let’s say labeling each of these data points takes about 30 seconds for the average employee, which amounts to around 800 hours of labor — at $10/hour, that costs you $8000.

The next step is to split your dataset into training data and test data. This should only take a few minutes if you have the technical skill to manipulate large datasets. The harder part is coming up with the appropriate evaluation criteria, which sets the standard for how your model should behave. The amount of time this takes varies with the problem you’re trying to solve. The evaluation criteria may be readily defined for some problems  for example, an accuracy score  while others might need something more complicated, like an AUC score.

Once you’ve set your evaluation criteria, you’ll need to do feature extraction to transform your labeled data into an appropriate form for training a model. You can then move on to the actual training of your model, which is an iterative process of refining the model’s performance with your test data. Finally, you’ll spend several weeks  or even months  packaging your model in a computer program that you can then use to draw insights from your data and help solve your business problem.

As you can see, it can take a long time to build a machine learning model. This is just one of the barriers to entry that you might face if you’re trying to do it in-house. Now, let’s take a closer look at some of the other barriers.


Barriers to entry for adopting machine learning in-house

Some barriers to entry become apparent after examining the process of building a model, while others may appear later as hidden costs:

    1. Lengthy construction time

      It often takes months – even years – to build a model, depending on the complexity of the problem you’re trying to solve.

 

    1. Laborious data collection and preparation

      We’ve outlined this in the workflow above, but to reiterate: collecting, organizing, and labeling data for your models can take weeks, if not months of valuable man-hours.

 

    1. Scarce and costly expertise

      Finding and funding a team of experts capable of building an in-house machine learning model for you is difficult and expensive. Did you know that as of 2014, the average base salary of a data scientist in the U.S. is $105,000? That’s not including benefits, bonuses, or any other possible compensation, which pushes that average up to $144,000. You’ll also need data engineers and probably a project manager with a technical background. In short, a full data science team isn’t something most startups can afford.

 

    1. Scalability and deployment difficulties

      Models can generally process information in a fraction of a second. However, processing time might increase depending on the complexity of the model. More complex models require a sophisticated infrastructure to handle large amounts of information at a high speed. Sometimes, processing will be further complicated by usage spikes, and scaling resources up and down in response to immediate demand is difficult and costly.

 

  1. Demanding upkeep

    Machine learning models require regular maintenance and testing with new data to ensure the model meets expectations. Methods used to train models are continuously improving, and you’ll need apply these techniques to maintain a competitive level of performance. Without a dedicated research team, it would be impossible to keep up with the latest technology.


Take a shortcut instead

Sure, anyone can deploy a model given enough time — but as they say, time is money. You’ll want to get value from your data projects as quickly as possible, and it’s hard to keep up with the pace of innovation. Luckily, you can skip over these hurdles by using a cloud-based machine learning service. These services provide pre-trained models – models that have been built ahead of time to complete general tasks, like analyzing the tone of a sentence.

By using a service, the expertise, time, and resources required to construct, scale, and deploy a machine learning model is all neatly packaged and available for a fraction of the cost and time it would take to create one yourself. You don’t have to worry about maintenance either: at indico, our team is always training our models on better data, improving the robustness and scalability of the infrastructure to support bigger, faster models.

That’s not all. Want to hear the best part? Any developer could easily use one of these pre-trained models to move you straight from your business problem to your business solution without hidden costs and wasted resources.

Using a Machine Learning Service vs. Building a Model In-house

Pre-trained models range in flexibility and customization  you can learn more about the various options available in our post that shows you how to choose the right machine learning service for your business.

Follow indico on Twitter or sign up for our blog newsletter to stay updated!

Questions? Email us at contact@indico.io.

[addtoany]

Increase intake capacity. Drive top line revenue growth.

[addtoany]

Unstructured Unlocked podcast

March 27, 2024 | E43

Unstructured Unlocked episode 43 with Sunil Rao, Chief Executive Officer at Tribble

podcast episode artwork
March 13, 2024 | E42

Unstructured Unlocked episode 42 with Arthur Borden, VP of Digital Business Systems & Architecture for Everest and Alex Taylor, Global Head of Emerging Technology for QBE Ventures

podcast episode artwork
February 28, 2024 | E41

Unstructured Unlocked episode 41 with Charles Morris, Chief Data Scientist for Financial Services at Microsoft

podcast episode artwork

Get started with Indico

Schedule
1-1 Demo

Resources

Blog

Gain insights from experts in automation, data, machine learning, and digital transformation.

Unstructured Unlocked

Enterprise leaders discuss how to unlock value from unstructured data.

YouTube Channel

Check out our YouTube channel to see clips from our podcast and more.
Subscribe to our blog

Get our best content on intelligent automation sent to your inbox weekly!