Azure Data Factory data source projection error

Consider the workaround Azure Data Factory projection error…

In this chit, I describe a workaround for the error of projecting data source schema in the Azure Data Factory.

Error description

  1. Add data source (Cosmos DB). Important note: because we have a NoSQL database, data schema could drift (different documents could have a different set of properties).
  2. In the source options tab set a query that enumerates all the possible properties of a document.
  3. Click Import projection button
set dataset fetch query

Expected result

We expect to see all the columns from the query in the projected schema.

Actual result

We see nothing (or only some of these columns, not all).

incorrect projection schema

Fix

To fix this Azure Data Factory projection error we can set the projection schema manually by editing the data flow script.

Click on the script button and find your data source definition

source(
	allowSchemaDrift: true,
	validateSchema: false,
	ignoreNoFilesFound: false,
	limit: 100,
	query: 'SELECT \nc.OrganizationName,\nc.FeatureDescription ...',
	format: 'documentQuery',
	systemColumns: false) ~> Features

As you can see a block with projection schema description is missed. So we need to add one.

source(
        output(
		OrganizationName as string,
		FeatureDescription as string
	),
	allowSchemaDrift: true,
	validateSchema: false,
	ignoreNoFilesFound: false,
	limit: 100,
	query: 'SELECT \nc.OrganizationName,\nc.FeatureDescription ...',
	format: 'documentQuery',
	systemColumns: false) ~> Features

And… that’s basically it. This simple step should resolve the problem.

Hope it helps you.


Read other Azure related articles by tag Azure

Leave a Reply