Skip to main content
If you’re new to Unstructured, read this note first.Before you can create a destination connector, you must first sign in to your Unstructured account:After you sign in, the Unstructured user interface (UI) appears, which you use to get your Unstructured API key.
  1. After you sign in to your Unstructured Let’s Go, Pay-As-You-Go, or Business account, click API Keys on the sidebar.
     For a Business account, before you click API Keys, make sure you have selected the organizational workspace you want to create an API key for. Each API key works with one and only one organizational workspace. Learn more.
  2. Click Generate API Key.
  3. Follow the on-screen instructions to finish generating the key.
  4. Click the Copy icon next to your new key to add the key to your system’s clipboard. If you lose this key, simply return and click the Copy icon again.
After you create the destination connector, add it along with a source connector to a workflow. Then run the worklow as a job. To learn how, try out the the notebook Dropbox-To-Pinecone Connector API Quickstart for Unstructured, or watch the two 4-minute video tutorials for the Unstructured Python SDK.You can also create destination connectors with the Unstructured user interface (UI). Learn how.If you need help, email Unstructured Support at support@unstructured.io.You are now ready to start creating a destination connector! Keep reading to learn how.
Send processed data from Unstructured to OpenSearch. The requirements are as follows.
  • For the Unstructured UI or the Unstructured API, local OpenSearch instances are not supported.
  • For Unstructured Ingest, local and non-local OpenSearch instances are supported. For example, to set up an Amazon OpenSearch Service managed cluster instance, complete steps similar to the following:
    1. Sign in to your AWS account, and then open your AWS Management Console.
    2. Open your Amazon OpenSearch Service console.
    3. On the sidebar, expand Managed clusters, and then click Dashboard.
    4. Click Create domain.
    5. In the Name tile, for Domain name, enter some unique domain name for your new OpenSearch instance.
    6. In the Domain creation method tile, select a method for creating the domain. For faster setup with sensible default settings, this example uses the Easy create method. Learn more about the Standard create method.
    7. In the Engine options tile, for Version, AWS recommends that you select the latest version.
    8. In the Network tile, for Network, select a network access method. For faster setup, this example uses the Public access method. Learn more about the VPC access method.
    9. For IP address type, select Dual-stack mode.
    10. In the Fine-grained access control tile, do one of the following:
      • If you want to use an existing AWS IAM user in the AWS account as the domain’s master user, then for Master user, select Set IAM ARN as master user. Then enter the IAM ARN for the master user in the IAM ARN box.
      • If you want to create a master user and password as the domain’s master user instead, then for Master user, select Create master user. Then specify some username and password for this new master user by filling in the Master username, Master password, and Confirm master password fields. Make sure to save the master user’s password in a secure location.
    11. Click Create.
    12. After the domain is created, you must allow Unstructured to access the domain, as follows: a. If the new domain’s settings page is not already showing, open it as follows: in your Amazon Open Search Service console, on the sidebar, expand Managed clusters, and then click Domains. Then, in the list of available domains, click the name of the newly created domain.
      b. On the Security configuration tab, click Edit.
      c. In the Access policy tile, for Domain access policy, select Only use fine-grained access control.
      d. Click Clear policy.
      e. Click Save changes.
    To set up an Amazon OpenSearch Serverless collection, complete steps similar to the following:
    1. Sign in to your AWS account, and then open your AWS Management Console.
    2. Open your Amazon OpenSearch Service console.
    3. On the sidebar, expand Serverless, and then click Dashboard.
    4. Click Create collection.
    5. In the Collection details tile, for Collection name, enter some unique name for your new OpenSearch Serverless collection. Optionally, for Description, enter some meaningful description for your new collection.
    6. For Collection type, select Search.
      Unstructured does not support the Vector search collection type. If you need vector search support, you can either continue with these steps to use the Search collection type, or you can follow the preceding steps to set up set up an Amazon OpenSearch Service managed cluster instead. However, note that the Amazon OpenSearch Serverless Search collection type is not as optimal as the Vector search collection type.
    7. In the Collection creation method tile, select Standard create.
    8. For Encryption, choose an AWS KMS key type.
    9. For Network access settings, choose an Access type.
    10. For Resource type, select both Enable access to OpenSearch endpoint and Enable access to OpenSearch Dashboards.
    11. Click Next.
    12. In the Definition method tile, select JSON.
    13. In the JSON editor box, enter the following JSON, replacing the following placeholders:
      • Replace <collection-name> with the name of the new OpenSearch Serverless collection.
      • Replace <account-id> with the target AWS account ID.
      • Replace <user-id> with the ID of the target AWS IAM user.
      [
    {
        "Rules": [
            {
                "Resource": ["collection/<collection-name>"],
                "Permission": [
                    "aoss:CreateCollectionItems",
                    "aoss:UpdateCollectionItems",
                    "aoss:DescribeCollectionItems"
                ],
                "ResourceType": "collection"
            },
            {
                "Resource": ["index/<collection-name>/*"],
                "Permission": [
                    "aoss:CreateIndex",
                    "aoss:DescribeIndex",
                    "aoss:ReadDocument",
                    "aoss:WriteDocument",
                    "aoss:UpdateIndex",
                    "aoss:DeleteIndex"
                ],
                "ResourceType": "index"
            },
            {
                "Resource": ["model/<collection-name>/*"],
                "Permission": [
                    "aoss:DescribeMLResource",
                    "aoss:CreateMLResource",
                    "aoss:UpdateMLResource",
                    "aoss:DeleteMLResource",
                    "aoss:ExecuteMLResource"
                ],
                "ResourceType": "model"
            }
        ],
        "Principal": ["arn:aws:iam::<account-id>:user/<user-id>"]
    }
]
    14. Click Next.
    15. For Data access policy settings, select Create as a new data access policy.
    16. In the Name and description tile, enter some unique name and an optional description for the new data access policy.
    17. Click Next.
    18. Enter any desired index details, and click Next again. For example: a. For Index name, enter the name of the new index in the collection.
      b. For Automatic Semantic Enrichment fields, click Add, enter embeddings for Automatic Semantic Enrichment field name, click Add, and click *Confirm.
      c. For Lexical search fields, click Add, enter text for Field name and select Text for Data type, click Add, and click *Confirm.
    19. Click Submit.
    The following video shows how to set up a local OpenSearch instance.
  • The instance’s host URL, as follows:
    • For an Amazon OpenSearch Service instance, do the following:
      1. Sign in to your AWS account, and then open your AWS Management Console.
      2. Open your Amazon OpenSearch Service console.
      3. On the sidebar, expand Managed clusters, and then click Dashboard.
      4. In the list of available domains, click the name of your domain.
      5. In the General information tile, copy the value of Domain endpoint v2 (dual stack).
    • For an Amazon OpenSearch Serverless collection, do the following:
      1. Sign in to your AWS account, and then open your AWS Management Console.
      2. Open your Amazon OpenSearch Service console.
      3. On the sidebar, expand Serverless, and then click Dashboard.
      4. In the list of available collections, click the name of your collection.
      5. On the Overview tab, in the Endpoint tile, copy the value of OpenSearch endpoint.
    • For a local OpenSearch instance, see Communicate with OpenSearch.
  • The name of the search index on the instance. For the destination connector, if you need to create an index and you’re using a master user and password as the domain’s master user, you can use for example the following curl command. Replace the following placeholders:
    • Replace <host> with the instance’s host URL.
    • Replace <port> with the instance’s port number, which is typically 443 (for encrypted connections, and less commonly 9200 for unencrypted connections).
    • Replace <master-username> with the master user’s name, and replace <master-password> with the master user’s password.
    • Replace <index-name> with the name of the new search index on the instance.
    • Replace <index-schema> with the schema for the new search index on the instance. A schema is optional; see the explanation following this curl command for more information.
    curl --request PUT "<host>:<port>/<index-name>" \
--user "<master-username>:<master-password>" \
[--header "Content-Type: application/json" \
--data '<index-schema>']
    If you’re using an existing AWS IAM user as the domain’s master user instead, you should use the AWS Command Line Interface (CLI) to create the index instead of using the precedingcurl command. To learn how, see create-index in the AWS CLI Command Reference. For the destination connector, the index does not need to contain a schema beforehand. If Unstructured encounters an index without a schema, Unstructured will automatically create a compatible schema for you before inserting items into the index. Nonetheless, to reduce possible schema compatibility issues, Unstructured recommends that you create a schema that is compatible with Unstructured’s schema. Unstructured cannot provide a schema that is guaranteed to work in all circumstances. This is because these schemas will vary based on your source files’ types; how you want Unstructured to partition, chunk, and generate embeddings; any custom post-processing code that you run; and other factors. For objects in the metadata field that Unstructured produces and that you want to store in an OpenSearch index, you must create fields in your index’s schema that follows Unstructured’s metadata field naming convention. For example, if Unstructured produces a metadata field with the following child objects: 
    "metadata": {
  "is_extracted": "true",
  "coordinates": {
    "points": [
      [
        134.20055555555555,
        241.36027777777795
      ],
      [
        134.20055555555555,
        420.0269444444447
      ],
      [
        529.7005555555555,
        420.0269444444447
      ],
      [
        529.7005555555555,
        241.36027777777795
      ]
    ],
    "system": "PixelSpace",
    "layout_width": 1654,
    "layout_height": 2339
  },
  "filetype": "application/pdf",
  "languages": [
    "eng"
  ],
  "page_number": 1,
  "image_mime_type": "image/jpeg",
  "filename": "realestate.pdf",
  "data_source": {
    "url": "file:///home/etl/node/downloads/00000000-0000-0000-0000-000000000001/7458635f-realestate.pdf",
    "record_locator": {
      "protocol": "file",
      "remote_file_path": "file:///home/etl/node/downloads/00000000-0000-0000-0000-000000000001/7458635f-realestate.pdf"
    }
  },
  "entities": {
    "items": [
      {
        "entity": "HOME FOR FUTURE",
        "type": "ORGANIZATION"
      },
      {
        "entity": "221 Queen Street, Melbourne VIC 3000",
        "type": "LOCATION"
      }
    ],
    "relationships": [
      {
        "from": "HOME FOR FUTURE",
        "relationship": "based_in",
        "to": "221 Queen Street, Melbourne VIC 3000"
      }
    ]
  }
}
    You can adapt the following index schema example for your own needs. Note that outside of metadata, the following fields are required by Unstructured whenever you create your own index schema:
    • element_id
    • record_id, which is required by Unstructured for intelligent record updates.
    • type, which is not required, but highly recommended.
    • text
    • embeddings if embeddings are generated; make sure to set dimension to the same number of dimensions as the embedding model generates.
    {
  "settings": {
    "index": {
      "knn": true,
      "knn.algo_param.ef_search": 100
    }
  },
  "mappings": {
    "properties": {
      "element_id": {
        "type": "keyword"
      },
      "record_id": {
        "type": "text"
      },
      "text": {
        "type": "text"
      },
      "type": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "embeddings": {
        "type": "knn_vector",
        "dimension": 1536
      },
      "metadata": {
        "properties": {
          "is_extracted": {
            "type": "boolean"
          },
          "coordinates-points": {
            "type": "float"
          },
          "coordinates-system": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "coordinates-layout_width": {
            "type": "long"
          },
          "coordinates-layout_height": {
            "type": "long"
          },
          "filetype": {
            "type": "keyword"
          },
          "languages": {
            "type": "keyword"
          },
          "page_number": {
            "type": "integer"
          },
          "image_mime_type": {
            "type": "keyword"
          },
          "filename": {
            "type": "keyword"
          },
          "data_source-url": {
            "type": "keyword"
          },
          "data_source-record_locator-protocol": {
            "type": "keyword"
          },
          "data_source-record_locator-remote_file_path": {
            "type": "keyword"
          },
          "entities-items": {
            "properties": {
              "entity": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              },
              "type": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              }
            }
          },
          "entities-relationships": {
            "properties": {
              "from": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              },
              "relationship": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              },
              "to": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              }
            }
          }
        }
      }
    }
  }
}
    See also:
  • For non-local OpenSearch instances, or if you’re using basic authentication to a local OpenSearch instance, the master user’s name and password.
  • For local OpenSearch instances, if you’re using certificates for authentication instead of basic authentication:
    • The path to the Certificate Authority (CA) bundle, if you use intermediate CAs with your root CA.
    • The path to the combined private key and certificate file, or
    • The paths to the separate private key and certificate file.
    To learn more, see Authentication backends, HTTP basic authentication, and Client certificate authentication.
To create an OpenSearch destination connector, see the following examples. 
import os

from unstructured_client import UnstructuredClient
from unstructured_client.models.operations import CreateDestinationRequest
from unstructured_client.models.shared import CreateDestinationConnector

with UnstructuredClient(api_key_auth=os.getenv("UNSTRUCTURED_API_KEY")) as client:
    response = client.destinations.create_destination(
        request=CreateDestinationRequest(
            create_destination_connector=CreateDestinationConnector(
                name="<name>",
                type="opensearch",
                config={
                    "hosts": ["https://<host>:<port>"],
                    "index_name": "<index-name>",
                    "username": "<username>",
                    "password": "<password>",
                    "aws_access_key_id": "<aws-access-key-id>",
                    "aws_secret_access_key": "<aws-secret-access-key>",
                    "aws_session_token": "<aws-session-token>",
                    "use_ssl": <True|False>
                }
            )
        )
    )

    print(response.destination_connector_information)
Replace the preceding placeholders as follows:
  • <name> (required) - A unique name for this connector.
  • https://<host>:<port> (required) - The OpenSearch instance’s host URL, which typically takes the form of https://<host>:<port>.
  • <index-name> (required) - The name of the search index on the instance.
  • <username> - If you’re using basic authentication to the instance, the domain’s master user’s name.
  • <password> - If you’re using basic authentication to the instance, the domain’s master user’s password.
  • <aws-access-key-id> - If you’re using an existing AWS IAM user as the domain’s master user, the AWS access key ID for the AWS IAM user. If you’re also using AWS STS for authentication, this will be a temporary AWS access key ID.
  • <aws-secret-access-key> - If you’re using an existing AWS IAM user as the domain’s master user, the AWS secret access key for the AWS IAM user. If you’re also using AWS STS for authentication, this will be a temporary AWS secret access key.
  • <aws-session-token> - If you’re using AWS STS for authentication, the temporary AWS STS session token.
    AWS STS credentials (consisting of a temporary AWS access key, temporary AWS secret access key, and temporary AWS STS session token) can be valid for as little as 15 minutes or as long as 36 hours, depending on how the credentials were initially generated. After the expiry time, the credentials are no longer valid and will no longer work with the corresponding OpenSearch connector. You must get a new set of AWS STS credentials to replace the expired ones by, which produces a new, refreshed temporary AWS access key, temporary AWS secret access key, and temporary AWS STS session token. For more information, see Request temporary security credentials.After you generate refreshed temporary AWS STS credentials, you must update the OpenSearch connector’s settings with the new, refreshed AWS STS credentials.
  • <field-name> (source connectors only) - Any specific fields to be accessed in the index.
  • <use-ssl> (required) - True if the OpenSearch instance requires an SSL connection; otherwise, false.