Uncategorized Archives - Collective Intelligence

Advanced Analytics Centers of Excellence

Dave Brener — Mon, 12 Dec 2022 20:54:38 +0000

To maximize the ROI of Advanced Analytics (AA) the enterprise must design and deploy an organizational structure that supports its mission.

As companies recognize the predictive power of advanced analytics, many are hoping to use AA to drive business decisions and strategies. Many of my clients have made AA a key component of their digital transformations.

While most companies understand the importance of analytics and have adopted best practices, fewer than 20 percent, according to a recent Forrester survey, maximize the potential and achieved AA at scale.

Because of this, most companies are frustrated as they see their efforts not meeting their goals and objective and watch as more mature, analytically driven companies leverage their enterprise data. The democratization of data has suddenly blurred sector boundaries and because of this, businesses will find themselves disrupted not by the competitor that they have been monitoring, but by an unknown upstart from another industry. Industry leadership is no longer enough; companies must aim to be at par (or better) across industries to compete effectively and ensure survival. Functional expertise, beyond specific business sector expertise, will become more and more vital. I believe that AA performance is achievable only by developing functional expertise, strategic partnerships, and defining clear objectives for organizing human capital.

Companies attempting an AA transformation can incorporate these elements into one of several organizational models assuming that clear governance is in place and the company fosters an analytics driven culture across all business units.

The three organizational models are:

Centralized — the company creates a single AA organization that stands alone in a Center of Excellence (COE) that supports all business units.
Decentralized — analytics expertise is embedded within individual business units.
Hybrid — combine centralized AA with embedded analytics expertise in some business units.

All organizations change over time, particularly as the AA transformation evolves. Many companies move back and forth between centralized and decentralized models ultimately settling on a hybrid strategy.

Data governance, however, should be centralized, even if data ownership/stewardship is not. Most large organizations have data centralized within business units like marketing, sales, operations, and finance.

Another important human capital decision is whether AA-COE talent should be partially outsourced and if so, what guidelines should be established. Outsourcing may be limited to low-level data analytics activities, but successful companies establish tactical and strategic partnerships to help with both tactical and strategic roles. These strategic relationships, managed by a single COE unit, provide the COE with guidance, mentorship and skill transfer. Many of the companies I work with protect certain analytical subject areas that produce a competitive advantage — such as pricing analytics — and staff solely within the organization.

Most organizations centralize partnership management in order to remove the likelihood of creating redundant or competing partnerships thereby risking efficiency and security.

So where does the AA unit live in the larger organizational hierarchy? AA is most valuable when it focuses cross-functionally and is accessible from all business units. To be most effective the COA should be visible from, and have access to, the C-suite. A COA with a bi-directional enterprise view will have the most transformational potential.

Over time I’ve seen most COEs collocated as sub-units of Business Intelligence. Many of our customers locate their AA units in IT, but this arrangement can be challenging. IT staff accustomed to managing longer-term projects that are often disconnected from the business, may not be prepared to manage short-term, agile AA projects.

Centralized Organizational Structure

Figure 1 illustrates the ideal organization structure for constructing a centralized Advanced Analytics Center of Excellence. Several job roles that traditionally were thought to be decentralized and distributed across other functional groups are added to this hierarchy in order to achieve maximum efficiency.

Let’s take a deeper dive into the key roles and responsibilities within an Advance Analytics Development trunk of the Centralized COE depicted within Figure 1.

Centralized COE Roles

Role: AA Program and Project Management

For the sake of brevity, I won’t define program/project management but it’s important to point out that those roles need to be cultivated within the COA because, quite frankly, AA projects need to be approached a bit differently than your typical IT project.

Because of the exploratory nature of AA projects, they’re often more similar to R&D efforts than traditional development efforts. R&D is challenging to plan, track and manage.

Moreover, AA projects can reap the benefits of an agile methodology. Agile Advanced Analytics (AAA) business processes should be adopted in order to reduce the time it takes for AA to return value to the organization while quickly adapting to change.

Role: Data Scientist

The job of data scientist is threefold:

Enhancing business intelligence by taking data that the enterprise collects and getting it in front of the right stakeholders in the form of dashboards, reports, other visualizations
Decision science, which takes enterprise data and uses it to help a company make a decision
Machine learning, which builds predictive data science models and places them into production

There is a bit of overlap here with the BI Analyst role however, digging a bit deeper, we can classify data science activities into two categories:

Type 1 — statistical analytics, or using a background in statistics or actuarial science to construct decision support assets
Type 2 — machine learning, using AI frameworks to build autonomous/semi-autonomous decision support assets.

The Data Scientist role also overlaps somewhat with that of the Data Engineer’s. Data Scientists frequently transform data within statistical analysis or machine learning experiments in order to perform feature engineering tasks like clustering or normalization.

There are many tools and technologies that facilitate data science activities some of which are:

Computer languages: R, Python, Java/Scala, Octave, and MATLAB
DBMS: SQL/NOSQL, Hadoop, Spark, and many OEM versions from companies like Databricks and Snowflake
Machine learning pipelines: tensor flow, AZURE ML Studio, Keras, etc.,
Platforms: SAS, Watson Analytics

Role: Solution Architect

IT Architecture is a well-established discipline that links business strategy, objectives and constraints into a viable, robust and cost-effective implementation plan. It is essential for ensuring solutions meet current requirements and can evolve to support future requirements without costly rework and disruptions.

An AA Solution Architect leverages this discipline to deal with the complex aspects of information management and advanced analytics while focusing on activities from user experience and performance to security and governance to platform and infrastructure.

The solutions architect has a deep understanding of the enterprise information model, the enterprise application portfolio, the enterprise technology infrastructure environment and has a deep understanding of the current advanced analytics technology landscape.

Key responsibilities of the analytics architect include operationalizing analytics, mapping business requirements to implementation approaches, selecting technology, and evaluating non-functional attributes such as usability, security, governance, and stability.

Role: AA Business Analyst

A Business Analyst combine analytics knowledge with strong domain expertise. In many organizations, within the context of advanced analytics, the business analyst might also be known as an operations research analyst or a business data analyst.

Within the AA COE Business Analyst duties typically include:

Evaluating business processes and the data they produce to meet the reporting and business intelligence needs of the users they support.
Develop business intelligence and advanced analytics use cases.
Communicating discoveries/insights with business teams, key stakeholders and the AA development team.
Preparing strategic recommendations for tracking metrics, KPIs and the deployment of business intelligence and advanced analytic assets.
Define how the business will make decisions based on the BI and advanced analytics assets that will be delivered.

The AA Business Analyst must have a working knowledge of the technology involved in analytics platforms used within the enterprise, though the need for hard technical skills is generally lower than for the BI Developer.

Role: BI Developer

Business Intelligence (BI) Developers transform data into insights that drive business value. Through use of data analytics, data visualization and data modeling techniques and technologies, BI analysts identify trends that help other departments, managers and executives make business decisions to modernize and improve processes in the organization.

BI developers typically are chartered to uncover improvements that can be made to save the enterprise money or increase profits. Working with business stakeholders they help define KPIs (Key Performance Indicators), Scorecards and Dashboards. This is done by mining complex data using BI software and tools, comparing data to competitors and industry trends and by creating visualizations that communicate findings to others in the organization. BI Analysts are proficient in computer programming languages (Python, R, Java, Scala), BI tools (Power BI, SAS, Tableau, Excel), and underlying technologies (RDBMS, NOSQL, Data Lakes).

Job responsibilities vary by organization, but the following day-to-day tasks are performed by BI Developers:

Review and validate enterprise data as it’s collected and stored.
Model data
Oversee the deployment of data to the data warehouse
Review enterprise data to ensure integrity/quality of data warehoused
Develop policies and procedures for the collection, analysis, and distribution of data
Design, create and monitor analytics assets
Monitor analytics consumption within the enterprise
Implement new data analysis methodologies, techniques, and tools

Role: Data Engineer

Data engineers prepare and transform data using pipelines. This involves extracting data from various data source systems, transforming it into a staging area or staged state, and loading it into a data warehouse system. This process is known as ETL (Extract, Transform, Load). Data engineers are also experienced in ELT (Extract Load Transform) technologies that delegate the transformation of the data to the underlying data management technology.

Data engineers are typically responsible for finding and analyzing patterns in datasets. This requires transforming large amounts of data into formats that can be processed and analyzed. The role of a data engineer requires significant technical skills, including multiple programming languages, ETL/ELT tools and knowledge of SQL, NOSQL, Big Data and Data Lake technologies.

Data engineers are expected to:

Create and maintain optimal data pipeline architecture
Assemble large, complex data sets that meet business/analytic requirements
Optimize data delivery and re-design infrastructure for greater scalability
Be conversant in data manipulation technologies like SQL, Big Query, Hadoop/Spark or other modern, cloud-scale data management technologies (Databricks, Snowflake)
Build the infrastructure required for optimal extraction, transformation, and loading of data from a wide variety of data sources using EDW/Data Lake technologies
Build the analytics infrastructure that creates the data pipelines that provide raw data in formats that can be used by BI Analysts and Data Scientists
Work with internal and external stakeholders to assist with data-related technical issues (data quality) and support data infrastructure needs
Understand and implement DevOps processes that supports the high availability of the EDW and Data Lake.

Role: Data Acquisition Manager

In order to understand the role of the Data Acquisition Manager we should first define what we mean by data acquisition.

Data acquisition is the processes for bringing data that has been created by a source outside the organization, into the organization, for production use.

Prior to the Big Data revolution, companies were predominantly inward-looking in terms of data consumption. Traditional, data-centric environments like data warehouses dealt only with data created within the enterprise.

With the advent of data science and predictive analytics, many organizations have come to the realization that enterprise data must be fused with external data to enable and scale a digital business transformation. This means that processes for identifying, sourcing, understanding, assessing and ingesting such data must be developed.

This highlights two points of terminological confusion. First, “data acquisition” is sometimes used to refer to data that the organization produces, rather than (or as well as) data that comes from outside the organization. This is a fallacy, because the data the organization produces is already acquired and likely warehoused in one or more data stores.

Second, the term “ingestion” is often used in place of “data acquisition.” Ingestion is merely the process of copying data from outside an environment to inside an environment and is very much narrower in scope than data acquisition. It seems to be a term that is more commonplace, because there are mature ingestion tools in the marketplace. (These are extremely useful, but ingestion is not data acquisition.) In fact, ingestion is the primary objective of the data engineer.

The following set of tasks constitute a data acquisition process and are overseen by the Data Acquisition Manager:

A need for externally sourced data is identified, perhaps with use cases
Prospecting for the required data is carried out
Data sources are disqualified, leaving a set of qualified sources
Vendors providing the sources are contacted and legal agreements entered into for evaluation and sample data sets are acquired for evaluation
Semantic analysis of the data sets is undertaken, so they are adequately understood
The data sets are evaluated against originally established use cases
Legal, privacy and compliance issues are understood, particularly with respect to permitted use of data
Vendor negotiations occur to purchase the data
Implementation specifications are drawn up, usually involving Data Operations who will be responsible for production processes
Source onboarding occurs, such that ingestion is technically accomplished
Production ingest is undertaken

Several things that stand out about this list. The first is that it consists of a relatively large number of multidisciplinary tasks. The second is that many different groups will be involved with the process.

Business Analysts, BI Analysts and Data Scientists will likely identify the need and document the use cases, whereas Data Governance, and perhaps the Office of General Counsel, will render opinions on legal, privacy and compliance requirements.

Conclusion

Creating a Center of Excellence (COE) will ensure that your organization reaps all the advantages of Advanced Analytics (AA) technologies. In this article I’ve identified a number of best practices that will help you kickstart efforts to build an AA driven culture, an organizational structure supporting that culture and the role human capital plays within the COE.

The post Advanced Analytics Centers of Excellence appeared first on Collective Intelligence.

Have You Considered the Data Mesh?

Dave Brener — Mon, 12 Dec 2022 20:28:16 +0000

Data mesh is a data management approach that emphasizes decentralized ownership and governance of data. It is designed to address the challenges of traditional data management approaches, which often rely on a centralized model with strict hierarchies and rules for accessing and using data.

The data mesh methodology is based on several key principles and practices, including:

Decentralization: Data mesh emphasizes decentralized ownership and governance of data, allowing for more open and collaborative access to data across the organization. This can foster greater innovation and collaboration, and ultimately drive business value.
Domain-driven design: Data mesh promotes the use of domain-driven design, which focuses on understanding and aligning data with the business domains it serves. This can help organizations better understand the value and use of their data and make more informed decisions about how to manage and use it.
Data governance: Data mesh advocates for a more flexible and agile approach to data governance, which can support decentralized ownership and use of data. This can include practices such as data cataloging, data lineage, and data governance boards to support collaboration and innovation.
Culture and mind shift: Data mesh recognizes that cultural and organizational change is key to successful data management. It emphasizes the importance of fostering a culture of data literacy and empowerment and encourages organizations to shift their mindset to prioritize data as a strategic asset.

In summary, the data mesh methodology is a data management approach that emphasizes decentralized ownership and governance of data, and promotes practices such as domain-driven design, data governance, and cultural change to drive business value.

The post Have You Considered the Data Mesh? appeared first on Collective Intelligence.

Becoming a Data Scientist – Your 10 Step Plan

Dave Brener — Mon, 12 Dec 2022 20:17:52 +0000

As a data scientist, you need to have a wide range of skills and knowledge to be successful. Here are the top ten things to learn to become a data scientist:

Programming languages: Data scientists typically use programming languages such as Python and R to manipulate, analyze, and visualize data. It’s important to learn the basics of these languages and become proficient in using them for data analysis tasks.
Statistics and probability: Data science involves working with large amounts of data, and a strong foundation in statistics and probability is essential for making sense of that data. This includes understanding concepts such as descriptive statistics, hypothesis testing, and probability distributions.
Machine learning: Machine learning is a key part of data science, and it involves using algorithms and models to automatically learn from data and make predictions or decisions. Learning about different types of machine learning algorithms, such as supervised and unsupervised learning, is essential for a data scientist.
Data visualization: Data visualization is the process of creating graphical representations of data, and it is an important tool for communicating findings and insights to others. Learning how to use data visualization tools and techniques, such as plots, charts, and maps, is crucial for a data scientist.
Data cleaning and preparation: Raw data is often messy and unstructured, and it requires significant cleaning and preparation before it can be used for analysis. Learning how to identify and handle missing or invalid data, and how to transform and merge data from multiple sources, is an essential skill for a data scientist.
Databases, data lakes and SQL: Data scientists often need to work with large datasets that are stored in databases and data lakes, and they use SQL (Structured Query Language) to query and manipulate that data. Learning SQL is important for a data scientist, as it allows them to access and work with data in a database.
Big data technologies: Data science often involves working with very large datasets, and big data technologies, such as Hadoop and Spark, are designed to handle and process that data efficiently. Learning about these technologies and how to use them can be beneficial for a data scientist.
Domain expertise: Data scientists often work on problems and projects in specific domains, such as finance, healthcare, or retail. It’s important for a data scientist to have some knowledge and understanding of the domain they are working in, as this can help them make better predictions and insights from the data.
Communication and collaboration: Data scientists often work on teams, and it’s important for them to be able to effectively communicate and collaborate with others. This includes being able to clearly explain technical concepts to non-technical stakeholders, and to work with other data scientists and analysts to share knowledge and insights.
Continuous learning: The field of data science is constantly evolving, and it’s important for data scientists to stay up-to-date with the latest developments and techniques. This involves continuously learning and acquiring new skills, such as learning new programming languages or machine learning algorithms, to stay ahead in the field.

Feel free to reach out to me if you’d like to discuss this. You can find me on Twitter @cichuck or on LinkedIn.

Cheers and best of luck!

The post Becoming a Data Scientist – Your 10 Step Plan appeared first on Collective Intelligence.