SharePoint is a huge platform and sometimes we may have to do some tricks to achieve our requirements. I was in a need to make my scanned PDF content to be searchable in the SharePoint online, which I have already achieved in a way, you can see that article here. Please consider this article as the second part of the above-mentioned article. Here in this article, we will make the Scanned PDF and images contents to be searchable in SharePoint online using the new Azure Form Recognizer and Microsoft Flow. Please keep reading.

Background

In our previous article, we learned how to make the Scanned PDFs to be searchable by its contents using the technology called OCR with a third party tool AquaForest. AquaForest is really a cool product and there are many things that you can do, but it is expensive, as I was using that just for the OCR purpose, it was not worth the money I spend. Because of that, I had to find a different option to satisfy my requirements, that is how the Azure Form Recognizer comes into this story. If you have ever used the Azure Computer Vision AI, you can see that there we use OCR to read the content of the image files, unfortunately, that doesn’t work well with PDF files. The Azure Form Recognizer removes that limitation.

Azure Form Recognizer is part of the Cognitive Services Family, if you are new to Cognitive Service, please feel free to read some of my articles on the same topics.

Update the Document Library List

As you all know that the SharePoint search will work with the content of the list and the metadata. So my idea here is to create a new column Metadata in the Document library list and then Azure Form Recognizer result to this field so that we search with the content this list entry, that is, our Scanned PDF will be available in the search result. Don’t worry if it sounds too complex, in fact, it is way too easy.

The reason why I am using a separate column here is to get the full control over the column and to set the Multiline support and allow unlimited length.

Creating the Flow to make the scanned PDF/Image contents to be searchable

Setting up Azure Form Recognizer

Now we need to create an Azure Form Recognizer, it is as simple as you create any other services in Azure. Go to the Azure Portal and search for the Form Recognizer, and create one.

Train your Form Recognizer Model

Now it is time to train our model so that the Form Recognizer can give us the appropriate output. You can do this step either by using the Web UI Console given by Microsoft or Curl.

If you are running the commands in Windows 10, run it with Bash or use the Invoke-WebRequest in PowerShell.

Where to run commands?

Using the web console is very easy so, I will use that. Before we run that we need to upload our sample document to the Azure Blob. Let’s do that now.

Configure Azure Storage Account and Upload Blob

Creating a storage account is really straightforward, search for Storage Account in the portal, and then fill the form as per your wish. Once you create the account, go to the resource and click on the Containers, under the Blob service menu. Now we need to create a container so that we can save the sample blob files inside. For now, I created this container with the name “models”. And then I uploaded 2 PDF files.

As mentioned here in the Microsoft Doc, now we need to get the SAS URL with our container name in it. Go to the Settings menu and click on the Share access signature and then click on the Generate SAS and connection string.

Now copy the Blob Service SAS URL and add your container name to the URL right after the windows.net/ so the end SAS URL will be looking like the below URL.

https://mlfit.blob.core.windows.net/models?sv=2019-02-02&ss=bfqgt&srt=shco&sp=rwdlhfacup&se=2020-03-05T16:51:40Z&st=2020-03-05T08:51:40Z&spr=https&sig=MSN0%2BhGHDGSDGW7jH2tOTGwh8I%2Bld%2BvcYAYTFGDSGH6mUyzsCAQXVoo%3D

Here the “models” is my container name.

Train your model

Now go to the console and fill all the details as below.

Here the resource name is the name of your Form Recognizer resource, Ocp-Apim-Subscription-Key is the key of that service, you can use key1 or key2. And in the Request body, edit the source property with your SAS URL. Then hit the send button. Now if everything goes well, you should get the output as below.

Please make a note of the modelId from the result, as we will use this in our Flow. If you are getting the error ” No valid blobs found in the specified Azure blob container”, then most probably it is because of the source filter we apply in the Request body, just remove that and hit the send button again.

{
  "error": {
    "code": "2024",
    "innerError": {
      "requestId": "78df3a9b-ae2c-47a7-900c-8fa78f5a5a15"
    },
    "message": "No valid blobs found in the specified Azure blob container."
  }
}

You can also try out the different API calls like “Get Models”, “Get Model”, etc. As that is not relevant to this article, I am not going to try them. The one thing to notice here is that the more you train, the more accurate the result will be.

Set up the Flow

If you are not sure about how you can create a flow, please look at the “Create a Flow” section here. Once you have the basic flow with the connector “When a file is created”, we can initialize our variables which we are going to use later.

One variable is to save the content type of the file we get and the other is to save the result of the Form Recognize Analyze API. Let’s move on to the next step now.

As you can see that the Flow is just like the programming tasks we do, we can use if, if-else, switch and many more. Try out these functions in your Flow when you get time.

So now we have a dynamic value in our contenttype variable. Let’s add the Analyze Form task, just search for the “Form Recognizer” and then select the action Analyze Form.

Now it will ask for you to enter the key of your Azure Form Recognizer, and a connection name. Once you give that, you can paste your modelId you got from the Train Model API call. In the end, this is how your Analyze Form action will look like.

You can see that we use the contenttype string variable here. Now we can append the value of the Analyze Form result to our recogoutput variable. And in the next step, we can save this information to the Metadata field that we created earlier for the Document library. Sounds good?

Now we are done with the Flow creation and if you wish you can send mails to the people about this conversion from the Scanned PDF to the Searchable PDF.

Below are the full steps of my flow.

Now we can test our flow. Cool right?

Test the Flow

To test, add a PDF document and an image to your Document Library and the flow will be triggered automatically. You can see the running status from the portal. Once the flow is run, you can see the result like below.

Now just go back to your Document library and see the data in the Metadata column, this will be the Body data that we get from the Analyze Form action. Now the only thing pending is to do some search actions. Can’t wait to do that.

Now try out with as many PDFs and Images you can, I will leave this to you.

Conclusion

Thanks a lot for staying with me for a long time and reading this article. I hope now you have learned about

• creating Azure Form Recognizer
• using Azure Form Recognizer to read text from PDF and Images
• training Azure Form Recognizer
• using Azure Storage Account
• creating a flow in SharePoint online
• creating the steps in Flow
• use the connections in Flow
• send mails from Flow

If you have learned anything else from this article, please let me know in the comment section.

Follow me

If you like this article, consider following me, haha!.

• GitHub
• medium
• Twitter

Your turn. What do you think?

Thanks a lot for reading. Did I miss anything that you may think which is needed in this article? Could you find this post useful? Kindly do not forget to share your feedback.

Kindest Regards
Sibeesh Venu

Introduction

We all get stuck somewhere in our so-called “Programmer Life” for a small requirement. And I was stuck with such a requirement that the content of the PDF file uploaded to my SharePoint online is not searchable, however, the PDF I created manually from the Word document works fine. Let me tell you why!. Typically there are 3 kinds of PDF files.

1. Normal PDF: These are the files that you get from applications like Microsoft Word, Adobe tools, etc. The beauty of this file is that the content of this file can be searched, you can select the text in this file, style them and copy-paste, etc.
2. Scanned PDF: This one is exactly opposite to the first one, and this was Villain in my requirement. The issue with this type is that though the content looks visually the same, it can not be searchable, select, copy-paste, etc, as in the end it is an image inserted to a PDF document. Now how can we read the contents of this file, that is where the technology called OCR (Optical Character Recognition) comes into the picture. With this, we can read the content, and make them searchable, etc. And when we do that, we introduce the third type of PDF file
3. Searchable/OCRed PDF: It is the type that we get from the OCR process as an output. In the end, this type will have two-layer in it, one is the image that we get from a scanner, and the second is the text content. With this two-layer, this file becomes almost equal to the first kind

Now let’s go see what was my requirement and how did I overcome this process.

Background

Technology is fast and starts running today if you want to touch it. I have a One Drive Sync folder to which I save the scanned PDF files from my scanner and once that is done the same will be synced to my SharePoint online. So far so good. But the problem is the content of these files are not searchable. Now let’s fix that.

Fix to make Scanned PDF files searchable

We use Microsoft Flow to do this process of converting the Scanned PDF to the Searchable PDF file. And in the flow, there are many ways that you can do this, I initially tried to do it with the combination of Computer Vision AI and some other services as preceding.

But, I was not getting the expected output when I was using them. So, I decided to go with other options. If you are new with OCR technology or Computer Vision AI, you can find my article here.

Create a flow

The files are being synced to my Document folder in SharePoint, thus I needed to create a flow that gets triggered whenever there is a file uploaded.

Click on the “Create a flow” then you will be asked to select the flow template. I selected the template “When a new file is added in SharePoint, complete a custom action”.

Once you click on the Continue button, you are good to create new steps in your flow.

Flow is a step by step solution and some steps may be having an output that we can carry to the next step and in our flow, we use this a lot. Once you connect to the SharePoint site, we need to get the uploaded file properties, to do that, click on the +(plus) icon, select “Add an action” and then search for “Get File Properties”

Now select the Site address and the library, and then click on the ID field, you will see an option to select the output of the previous step.

Now we get the file and need to check the file type right, to do that add a condition control and then add the conditions to it.

Each condition will have an output as “Yes” or “No” and in the “Yes” part, we will add all of our other steps and we will not think about the “No” output now. But, you can think of adding some tasks there.

Now in the “Yes” tab, we can get the file and pass it to the OCR process, that is where the tool called AquaForest comes into the story. Please follow the steps mentioned in this article and get the key needed. Once that is done, add the action “OCR PDF or Images” by searching the word ” AquaForest”.

Give the connection a name and add the key in the next popup. There are many properties that you can set here, but the below two are important.

As an output of this step, we get the OCRed file and now all we have to do is to add the action called “Create File” and set up the same.

Wow, now we have a Searchable PDF in our Document folder. Go search with any content of your newly updated PDF. If you wish, you can also create an action to send an acknowledgment mail.

Testing the flow

As we already created the flow, now it is time to test the same. To do that, I added a scanned document to my one drive folder. We can check the Flow running status in the portal.

Below is the sample run history output of my flow.

Conclusion

Thanks a lot for staying with me for a long time and reading this article. I hope now you have learned about

• creating a flow in SharePoint online
• creating the steps in Flow
• use the connections in Flow
• OCR the PDF using Computer Vision
• OCR the PDF using AquaForest API
• creating a new File with OCRed output
• send mails from Flow

If you have learned anything else from this article, please let me know in the comment section.

Follow me

If you like this article, consider following me, haha!.

• GitHub
• medium
• Twitter

Your turn. What do you think?

Thanks a lot for reading. Did I miss anything that you may think which is needed in this article? Could you find this post useful? Kindly do not forget to share your feedback.

Kindest Regards
Sibeesh Venu

Introduction

The Cognitive services are always a great topic to consider writing, especially when there is a mix of Machine Learning. If you are totally new to the Cognitive services, consider reading my posts wouldn’t be a bad idea. Here in this article, we are going to use the Microsoft Custom Vision AI, which is a cognitive AI service. With the help of Custom Vision AI, we can easily do the below things.

1. Upload image
2. Tag the images
3. Train the model
4. Evaluate the result

The Custom Vision service uses the machine learning to classify the images we are uploading. The only thing we are required to do is, just specifying the correct tag for each image, you can also tag thousands of images at a time. The AI algorithm is super powerful to give us better accuracy and once the model is trained, use the same to classify new images according to the needs of your app. Let’s skip the introduction and start doing some amazing things. Let’s just start then.

Background

I have a habit of writing article about the things I learn, though I had worked with Cognitive Services before, recently I had a chance to work with Custom Vision AI service, so thought of building something with that. Here in this article, we are going to create an Azure Custom Vision AI which can,

• Scan for real objects
• Identify whether the object is a car or not
• If that is a car, then check which is the car manufacturer

We will be creating a Custom Vision AI model, which we will train with thousands of images. I hope it helps.

Prerequisites

To do the wonderful things, we always need some prerequisites.

1. Account with Custom Vision AI, you can either use the free subscription or your Azure account
2. A database of images for training the model, the same can be downloaded from here
3. Good internet connection, as you will have to upload thousands of images
4. I have already collected a large dataset, which is more than enough to get started, you can download it from OneDrive here

Creating Your Custom Vision AI Model

I assume that you had already gone through the Prerequisites and followed the instructions. If not, please check again.

To get started here, please go to the https://customvision.ai/ and,

1. Log in with your credentials
2. Creating your project
3. Add as many relevant images possible and tag them appropriately
4. Once you tag them properly, you can train your model, if everything goes well, you should see an output as below.

Custom Vision AI Train Output

Please make sure that you are checking the precision, if it is a good number, then you are good to go. As many images you upload and train them with the correct tag, the precision value get increased.  So feel free to add images and see the output yourself.  Below is the count of the images which I trained to my model.

When you train again with the new data, you can see that a new iteration is getting created which is really handy if you need to go back to the previous state, if the precision value is getting decreased.

You can also do a quick test with the UI provided. Please click on the Quick test button and upload an image, let’s upload an image of a bedroom.

Bedroom Test

Well, that was easy. Now let’s upload an image of a Porsche car.

Porsche Car Result

Have you noticed the values returned? The custom vision API is saying that the image is 99.8% is a car and 99.3% is a Porsche car, which is really impressive. Now, what about BMW?

BMW Result

As our Custom Vision AI is working as expected, now we can go ahead and grab our keys which are needed when you create an application to use this Custom Vision AI. Please click on the setting symbol on the right side of the page, and note down your project id and prediction key. You will also need a prediction URL, which you can get from the Performance tab.

Conclusion

The possibilities of cognitive services and machine learning are limitless, you can think of using this in many of our real-life scenarios. If possible, please share me your thoughts on where are all the places we can use this Custom Vision AI service. In this article, we have learned,

• What is Custom Vision AI Service
• How to work with Custom Vision AI service
• How to train and test our Custom Vision AI model

Is that all? No, in the next article we will create an application which can be used to test it in real time.

Your turn. What do you think?

Thanks a lot for reading. Did I miss anything that you may think which is needed in this article? Could you find this post as useful? Kindly do not forget to share me your feedback. What are you waiting for? create your model and train.

Kindest Regards
Sibeesh Venu