SharePoint is a huge platform and sometimes we may have to do some tricks to achieve our requirements. I was in a need to make my scanned PDF content to be searchable in the SharePoint online, which I have already achieved in a way, you can see that article here. Please consider this article as the second part of the above-mentioned article. Here in this article, we will make the Scanned PDF and images contents to be searchable in SharePoint online using the new Azure Form Recognizer and Microsoft Flow. Please keep reading.

Background

In our previous article, we learned how to make the Scanned PDFs to be searchable by its contents using the technology called OCR with a third party tool AquaForest. AquaForest is really a cool product and there are many things that you can do, but it is expensive, as I was using that just for the OCR purpose, it was not worth the money I spend. Because of that, I had to find a different option to satisfy my requirements, that is how the Azure Form Recognizer comes into this story. If you have ever used the Azure Computer Vision AI, you can see that there we use OCR to read the content of the image files, unfortunately, that doesn’t work well with PDF files. The Azure Form Recognizer removes that limitation.

Azure Form Recognizer is part of the Cognitive Services Family, if you are new to Cognitive Service, please feel free to read some of my articles on the same topics.

Update the Document Library List

As you all know that the SharePoint search will work with the content of the list and the metadata. So my idea here is to create a new column Metadata in the Document library list and then Azure Form Recognizer result to this field so that we search with the content this list entry, that is, our Scanned PDF will be available in the search result. Don’t worry if it sounds too complex, in fact, it is way too easy.

The reason why I am using a separate column here is to get the full control over the column and to set the Multiline support and allow unlimited length.

Creating the Flow to make the scanned PDF/Image contents to be searchable

Setting up Azure Form Recognizer

Now we need to create an Azure Form Recognizer, it is as simple as you create any other services in Azure. Go to the Azure Portal and search for the Form Recognizer, and create one.

Train your Form Recognizer Model

Now it is time to train our model so that the Form Recognizer can give us the appropriate output. You can do this step either by using the Web UI Console given by Microsoft or Curl.

If you are running the commands in Windows 10, run it with Bash or use the Invoke-WebRequest in PowerShell.

Where to run commands?

Using the web console is very easy so, I will use that. Before we run that we need to upload our sample document to the Azure Blob. Let’s do that now.

Configure Azure Storage Account and Upload Blob

Creating a storage account is really straightforward, search for Storage Account in the portal, and then fill the form as per your wish. Once you create the account, go to the resource and click on the Containers, under the Blob service menu. Now we need to create a container so that we can save the sample blob files inside. For now, I created this container with the name “models”. And then I uploaded 2 PDF files.

As mentioned here in the Microsoft Doc, now we need to get the SAS URL with our container name in it. Go to the Settings menu and click on the Share access signature and then click on the Generate SAS and connection string.

Now copy the Blob Service SAS URL and add your container name to the URL right after the windows.net/ so the end SAS URL will be looking like the below URL.

https://mlfit.blob.core.windows.net/models?sv=2019-02-02&ss=bfqgt&srt=shco&sp=rwdlhfacup&se=2020-03-05T16:51:40Z&st=2020-03-05T08:51:40Z&spr=https&sig=MSN0%2BhGHDGSDGW7jH2tOTGwh8I%2Bld%2BvcYAYTFGDSGH6mUyzsCAQXVoo%3D

Here the “models” is my container name.

Train your model

Now go to the console and fill all the details as below.

Here the resource name is the name of your Form Recognizer resource, Ocp-Apim-Subscription-Key is the key of that service, you can use key1 or key2. And in the Request body, edit the source property with your SAS URL. Then hit the send button. Now if everything goes well, you should get the output as below.

Please make a note of the modelId from the result, as we will use this in our Flow. If you are getting the error ” No valid blobs found in the specified Azure blob container”, then most probably it is because of the source filter we apply in the Request body, just remove that and hit the send button again.

{
  "error": {
    "code": "2024",
    "innerError": {
      "requestId": "78df3a9b-ae2c-47a7-900c-8fa78f5a5a15"
    },
    "message": "No valid blobs found in the specified Azure blob container."
  }
}

You can also try out the different API calls like “Get Models”, “Get Model”, etc. As that is not relevant to this article, I am not going to try them. The one thing to notice here is that the more you train, the more accurate the result will be.

Set up the Flow

If you are not sure about how you can create a flow, please look at the “Create a Flow” section here. Once you have the basic flow with the connector “When a file is created”, we can initialize our variables which we are going to use later.

One variable is to save the content type of the file we get and the other is to save the result of the Form Recognize Analyze API. Let’s move on to the next step now.

As you can see that the Flow is just like the programming tasks we do, we can use if, if-else, switch and many more. Try out these functions in your Flow when you get time.

So now we have a dynamic value in our contenttype variable. Let’s add the Analyze Form task, just search for the “Form Recognizer” and then select the action Analyze Form.

Now it will ask for you to enter the key of your Azure Form Recognizer, and a connection name. Once you give that, you can paste your modelId you got from the Train Model API call. In the end, this is how your Analyze Form action will look like.

You can see that we use the contenttype string variable here. Now we can append the value of the Analyze Form result to our recogoutput variable. And in the next step, we can save this information to the Metadata field that we created earlier for the Document library. Sounds good?

Now we are done with the Flow creation and if you wish you can send mails to the people about this conversion from the Scanned PDF to the Searchable PDF.

Below are the full steps of my flow.

Now we can test our flow. Cool right?

Test the Flow

To test, add a PDF document and an image to your Document Library and the flow will be triggered automatically. You can see the running status from the portal. Once the flow is run, you can see the result like below.

Now just go back to your Document library and see the data in the Metadata column, this will be the Body data that we get from the Analyze Form action. Now the only thing pending is to do some search actions. Can’t wait to do that.

Now try out with as many PDFs and Images you can, I will leave this to you.

Conclusion

Thanks a lot for staying with me for a long time and reading this article. I hope now you have learned about

• creating Azure Form Recognizer
• using Azure Form Recognizer to read text from PDF and Images
• training Azure Form Recognizer
• using Azure Storage Account
• creating a flow in SharePoint online
• creating the steps in Flow
• use the connections in Flow
• send mails from Flow

If you have learned anything else from this article, please let me know in the comment section.

Follow me

If you like this article, consider following me, haha!.

• GitHub
• medium
• Twitter

Your turn. What do you think?

Thanks a lot for reading. Did I miss anything that you may think which is needed in this article? Could you find this post useful? Kindly do not forget to share your feedback.

Kindest Regards
Sibeesh Venu

Introduction

We all get stuck somewhere in our so-called “Programmer Life” for a small requirement. And I was stuck with such a requirement that the content of the PDF file uploaded to my SharePoint online is not searchable, however, the PDF I created manually from the Word document works fine. Let me tell you why!. Typically there are 3 kinds of PDF files.

1. Normal PDF: These are the files that you get from applications like Microsoft Word, Adobe tools, etc. The beauty of this file is that the content of this file can be searched, you can select the text in this file, style them and copy-paste, etc.
2. Scanned PDF: This one is exactly opposite to the first one, and this was Villain in my requirement. The issue with this type is that though the content looks visually the same, it can not be searchable, select, copy-paste, etc, as in the end it is an image inserted to a PDF document. Now how can we read the contents of this file, that is where the technology called OCR (Optical Character Recognition) comes into the picture. With this, we can read the content, and make them searchable, etc. And when we do that, we introduce the third type of PDF file
3. Searchable/OCRed PDF: It is the type that we get from the OCR process as an output. In the end, this type will have two-layer in it, one is the image that we get from a scanner, and the second is the text content. With this two-layer, this file becomes almost equal to the first kind

Now let’s go see what was my requirement and how did I overcome this process.

Background

Technology is fast and starts running today if you want to touch it. I have a One Drive Sync folder to which I save the scanned PDF files from my scanner and once that is done the same will be synced to my SharePoint online. So far so good. But the problem is the content of these files are not searchable. Now let’s fix that.

Fix to make Scanned PDF files searchable

We use Microsoft Flow to do this process of converting the Scanned PDF to the Searchable PDF file. And in the flow, there are many ways that you can do this, I initially tried to do it with the combination of Computer Vision AI and some other services as preceding.

But, I was not getting the expected output when I was using them. So, I decided to go with other options. If you are new with OCR technology or Computer Vision AI, you can find my article here.

Create a flow

The files are being synced to my Document folder in SharePoint, thus I needed to create a flow that gets triggered whenever there is a file uploaded.

Click on the “Create a flow” then you will be asked to select the flow template. I selected the template “When a new file is added in SharePoint, complete a custom action”.

Once you click on the Continue button, you are good to create new steps in your flow.

Flow is a step by step solution and some steps may be having an output that we can carry to the next step and in our flow, we use this a lot. Once you connect to the SharePoint site, we need to get the uploaded file properties, to do that, click on the +(plus) icon, select “Add an action” and then search for “Get File Properties”

Now select the Site address and the library, and then click on the ID field, you will see an option to select the output of the previous step.

Now we get the file and need to check the file type right, to do that add a condition control and then add the conditions to it.

Each condition will have an output as “Yes” or “No” and in the “Yes” part, we will add all of our other steps and we will not think about the “No” output now. But, you can think of adding some tasks there.

Now in the “Yes” tab, we can get the file and pass it to the OCR process, that is where the tool called AquaForest comes into the story. Please follow the steps mentioned in this article and get the key needed. Once that is done, add the action “OCR PDF or Images” by searching the word ” AquaForest”.

Give the connection a name and add the key in the next popup. There are many properties that you can set here, but the below two are important.

As an output of this step, we get the OCRed file and now all we have to do is to add the action called “Create File” and set up the same.

Wow, now we have a Searchable PDF in our Document folder. Go search with any content of your newly updated PDF. If you wish, you can also create an action to send an acknowledgment mail.

Testing the flow

As we already created the flow, now it is time to test the same. To do that, I added a scanned document to my one drive folder. We can check the Flow running status in the portal.

Below is the sample run history output of my flow.

Conclusion

Thanks a lot for staying with me for a long time and reading this article. I hope now you have learned about

• creating a flow in SharePoint online
• creating the steps in Flow
• use the connections in Flow
• OCR the PDF using Computer Vision
• OCR the PDF using AquaForest API
• creating a new File with OCRed output
• send mails from Flow

If you have learned anything else from this article, please let me know in the comment section.

Follow me

If you like this article, consider following me, haha!.

• GitHub
• medium
• Twitter

Your turn. What do you think?

Thanks a lot for reading. Did I miss anything that you may think which is needed in this article? Could you find this post useful? Kindly do not forget to share your feedback.

Kindest Regards
Sibeesh Venu

Introduction

I understand that you need to move all of your contents to your destination site from your source site in SharePoint online. You might have done some search in Google and saw this post. Here in this article, I will be walking you through the steps you need to perform the SharePoint PnP provisioning. With the latest SharePoint modern experience, Microsoft had decided to turn off some features, and one among them was Custom Scripting. Unfortunately, to do the SharePoint provisioning, we will have to enable the Custom Scripting. Enough talking, let’s jump into the coding. I hope you will find this post useful.

Before you start Provisioning your site

Before you jump into the process of provisioning there is some configuration which you must perform.

• Enable the Custom Scripting
• Set the User as a Term Store Admin

Enable the Custom Scripting

With the latest release, Microsoft had deactivated this feature from the modern SharePoint experience thus we would have to enable it manually. According to Microsoft, below are the features which will not be available until you manually enable the custom scripting.

1. Save Site as Template
2. Save document library as template
3. Solution Gallery
4. Theme Gallery
5. Help Settings
6. HTML Field Security
7. Sandbox solutions
8. SharePoint Designer
9. Uploading files that potentially include script

In our case, we wanted to do SharePoint PnP provisioning, and it is interesting that it is not mentioned anywhere that we need to enable custom scripting to do SharePoint PnP provisioning. You can always enable the Custom Scripting in two ways, one is using the portal, and another is to use PowerShell commands. I would prefer using the PowerShell commands as it will enable the custom scripting right away. If you enable the custom scripting via the portal, it will take 24 hours to get it activated.

To allow custom script on other SharePoint site collections, you will have to download the download the latest SharePoint Online Management Shell. Once you are done you can perform the below commands.

Connect-SPOService -Url https://yoursharepoint-admin.sharepoint.com

The above command will connect your SharePoint admin site, please be noted that you should have a tenant admin access to do this. If you don’t have the access, you should consider getting the access or you can request the tenant admin to do the same.

Once you are connected, you can perform the preceding command.

Set-SPOsite https://yoursharepinttenant.sharepoint.com/sites/yoursharepointsite -DenyAddAndCustomizePages 0

The above command will enable the custom scripting for the site you have mentioned. To check this, you can always go to any of your site collection/list settings, and if you see the option save the list as a template, it is done and you can proceed.

Save list as the template

Set the User as a Term Store Admin

Once we have enabled the custom scripting it is time to add you as the Term Store Admin, again you can always reach out to your Tenant admin to add you as a Term Store Admin. You can ask your Tenant admin to perform the preceding steps.

1. Sign in to Office 365 as a global admin or SharePoint admin.

2. Click on the app launch icon( 9 dots) and then click on Admin to open the Microsoft 365 admin center.
3. Select Admin centers and then click on SharePoint.
4. Now, you should see an option as Term Store and click on it.
5. He/She can add your mail id in the text box near to the heading Term Store Administrators

Once it is done you can always go to your Site Settings > Term Store Management > Term Store Administrators.

Start SharePoint PnP Provisioning

Wow!. We have done all the configuration and now we can start doing the provisioning.

Connect your source site

The first thing to do is, to connect the source SharePoint site, you can do that by using the preceding command.

Connect-PnPOnline https://youradmin.sharepoint.com/sites/yoursite/

Generate a provisioning template

Now we can generate a provisioning template from the web. You can do that by doing the command below.

Get-PnPProvisioningTemplate -Out c:/Provisioning/template.xml

Please be noted that this will create a new folder “Provisioning” in the C drive and then place the template.xml in it. You can always see the other options of Get-PnPProvisioningTemplate here. This command will pull almost everything from the source site, and you can always choose what exactly you need to take from the source site. This will not retrieve the data, you will have to use another command for that.

Add data to the provisioning template

As we have seen that the Get-PnPProvisioningTemplate will not retrieve the data with the list, we will have to use Add-PnPDataRowsToProvisioningTemplate command to do that magic.

Add-PnPDataRowsToProvisioningTemplate -Path c:/Provisioning/template.xml -List 'The list name' -Query '<view></view>'

Please see here to see the other options you can use with the Add-PnPDataRowsToProvisioningTemplate command. Please be noted that the size of the template.xml file will be larger as per the data you have in the source list.

Apply provision to your destination site

Now we have an XML file which has everything we need to move. Let’s just do it then. Connect your destination site by following the command below.

Connect-PnPOnline https://youradmin.sharepoint.com/sites/yourdetinationsite

Please remember to use the same id and password of your Term Store Admin account.

Once you are connected to the destination site, you can perform the Apply-PnPProvisioningTemplate as follows.

Apply-PnPProvisioningTemplate -Path C:/Provisioning/template.xml

If you are getting an error as ” Apply-PnPProvisioningTemplate : The parent content type specified by content type identifier does not exist”, you will have to enable the SharePoint Server Publishing feature in the Site Settings > Site Features. Remember to enable the SharePoint Server Publishing Infrastructure under Site Settings > Site Collection Features before you enable the SharePoint Server Publishing feature in the Site Settings. It is a dependency. You can always see my StackExchange answer here.

Apply-PnPProvisioningTemplate : The parent content type specified by content type identifier does not exist.
At line:1 char:1
+ Apply-PnPProvisioningTemplate -Path C:\Provisioning\template.xml
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : WriteError: (:) [Apply-PnPProvisioningTemplate], ServerException
    + FullyQualifiedErrorId : EXCEPTION,SharePointPnP.PowerShell.Commands.Provisioning.ApplyProvisioningTemplate

Please be noted that you will also get an error as “WARNING: ListInstance Style Library is a Style Library of a site with NoScript enabled and will be skipped” if the custom scripting is not enabled.

You will also get an error as “Access denied. You do not have permission to perform this action or access this resource” if you don’t have Term Store Admin permission.

Apply-PnPProvisioningTemplate : Access denied. You do not have permission to perform this action or access this resource.
At line:1 char:1
+ Apply-PnPProvisioningTemplate -Path C:/Provisioning/template.xml
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : WriteError: (:) [Apply-PnPProvisioningTemplate], ServerUnauthorizedAccessException
    + FullyQualifiedErrorId : EXCEPTION,SharePointPnP.PowerShell.Commands.Provisioning.ApplyProvisioningTemplate

Once you run the Apply-PnPProvisioningTemplate command, and if everything is fine, you will get everything in your destination site. It will take some time to generate everything for you and add the data to it, depends on the amount of the data you have the process can take from 5 minutes to hours.

Conclusion

Congratulation, we have successfully moved all of our data and almost everything from one SharePoint site to other. In this article, we have learned how to,

1. Enable Custom Scripting in SharePoint
2. Add a Term Site Admin
3. Connect to a SharePoint site
4. Create a PnP provisioning template from the web
5. Add data to your Provisioning template form the lists
6. Apply the provisioning to the destination SharePoint site

Your turn. What do you think?

Thanks a lot for reading. I will come back with another post on the same topic very soon. Did I miss anything that you may think which is needed? Could you find this post as useful? Kindly do not forget to share me your feedback.

Kindest Regards
Sibeesh Venu

Introduction

In this era of cloud computing and collaborative platform, SharePoint is a familiar name to the people of the business industry. The fact that more than 85% of Fortune 500 companies use SharePoint is the proof of its efficiency. If you are a business owner, then you should also consider using SharePoint to increase the productivity of your workforce. In this post, we will try to convey a clear concept of what is SharePoint and its benefits to the readers. First of all, let’s learn what is SharePoint and what it does.

What is SharePoint?

SharePoint is a product/ application used for collaboration, data storage, and document management. It also provides users a platform to develop web applications that can meet their customized requirements. All apps and data are stored in one place so that all members can conveniently access, store and share data. Cost reduction and risk management are two of the main reasons behind its popularity.

Based on the performing platform, there are two types of SharePoint: SharePoint on-premise and SharePoint Online. Now we will know the main attributes of SharePoint.

Attributes of SharePoint

There are some common attributes of SharePoint:

• Provides facilities to develop and deploy Share Point websites
• Creates user groups, types and lists
• Individual user rights can be defined
• Absolute security of user information
• Availability of site templates
• Allows customization of website UI
• Facilitates making and using blogs and wikis

Benefits & Advantages of SharePoint

The benefits of using SharePoint are numerous. For this discussion, we have chosen top 10 prime benefits of SharePoint.

1. Collaboration: SharePoint is highly praised for providing a perfect collaborative platform. All members or a team or organization can stay connected while working in partnership. Using the latest real-time streaming feature of SharePoint, sharing and accessing data within an organization have become faster than ever.
2. Customization: The need of every industry and organization is different from each other. SharePoint works relentlessly to serve all of them by keeping the individual demands in mind. At SharePoint, organizations get complete freedom to customize everything, from services to websites. From the large collection of intelligent tools, companies can pick according to their development needs and also customize them.
3. Website Unification: SharePoint has brought all sites of a company under one umbrella. It facilitates the consolidation of the Internet, intranet, and extranet sites. As a result, it costs the company less than the cost of separately done site management.
4. Full-proof Security: When it comes to security, SharePoint will get full marks. SharePoint uses SSL for trafficking user data that has been already encrypted. They never disclose the physical location of their server to anyone. According to policy, SharePoint data are never minded for advertising. SharePoint also provides data backup and protects user identity.
5. Centralized Administration and Storage: SharePoint offers storage facility to all data of your organization at one place. The entire platform can also be controlled by central administrators. Application management, data backup and restoration, upgradation, system settings configuration, everything can be controlled from one location, thanks to SharePoint.
6. Easy to Use and Learn: SharePoint has simplified the development process of business tools and sites. No need to hire a professional when even people without web development background can easily learn and apply the method of site development. This has actually shortened the response time to any immediate business need.
7. Utilization of Manpower and Expenditure: Earlier, IT employees hardly had any free time but with the introduction of SharePoint, things have changed. IT people do not have to spend time for maintenance, backup, and patching as these can be easily done by SharePoint Admin panel. As a result, IT staffs can utilize their time by doing something more productive. In this way, SharePoint ensures proper utilization of manpower and cost.
8. Anywhere Access to SharePoint Online: With the arrival of SharePoint Online that works in collaboration with Office 365, SharePoint users get complete freedom in terms of network and device. Now they can access SharePoint from literally anywhere. SharePoint also permits syncing its data with Windows Explorer. All the works done in offline mode get synced with the online version as soon as the device gets connected to Internet.
9. Easy Management of External Users: All SharePoint files and sites can be easily shared with anyone. Microsoft account holders can have easy access to SharePoint content. To share it with an external user, SharePoint generates guest links that do not require any sign-in. SharePoint allows you to set a policy regarding external sharing by which, the external access will be controlled systematically.
10. Flexible Scaling: SharePoint Online, which is a component of Office 365, allows flexible scaling of your organization. As an Office 365 subscriber, companies need to pay according to the number of employees. If there is an increase or decrease in the employee number, the monthly subscription plan helps to manage the cost. Now organizations can easily scale their farm without affecting the budget.

Conclusion

Here we have discussed what is SharePoint and its benefits. Readers will get a complete idea about the advantages of using SharePoint from this post. This write-up will also help users to decide whether to opt for this Office 365 product or not. SharePoint is a product trusted by millions of users worldwide. Business users can choose to use SharePoint if it meets their business requirements.