Repository navigation
json object not serializable for array of objects datatype #146
Description
Hi,
We have a created a custom resource provider type for aws using https://github.com/iann0036/cfn-tf-custom-types .
When trying to launch stack in aws, the datatypes with array of objects throwing an error as json not serializable.
Can some one help me out resolving the issue
Activity
ammokhov commentedon Dec 22, 2020
@sriram9707 could you please provide a bit more details:
sriram9707 commentedon Dec 23, 2020
@ammokhov i use the following versions.
cloudformation-cli==0.1.14
cloudformation-cli-python-plugin==2.1.2
python 3.8
awscli v2
sriram9707 commentedon Dec 29, 2020
@ammokhov is there any update on this please ?
johnttompkins commentedon Jan 4, 2021
Could you provide a stack arn
This doesn't look like a schema file. The schema should conform to the resource provider schema
If you could provide that schema here or provide a stack arn to jotompki@amazon.com we can take a look at that.
sriram9707 commentedon Jan 5, 2021
@jotompki Please find the resource schema here.
{ "typeName": "Terraform::AWS::Instance", "description": "Provides an EC2 instance resource. This allows instances to be created, updated,\nand deleted. Instances also support [provisioning](/docs/provisioners/index.html).", "sourceUrl": "https://github.com/iann0036/cfn-tf-custom-types.git", "documentationUrl": "https://github.com/iann0036/cfn-tf-custom-types/blob/docs/resources/aws/Terraform-AWS-Instance/docs/README.md", "definitions": { "CreditSpecification": { "type": "object", "additionalProperties": false, "properties": { "CpuCredits": { "type": "string" } }, "required": [] }, "EbsBlockDevice": { "type": "object", "additionalProperties": false, "properties": { "DeleteOnTermination": { "type": "boolean" }, "DeviceName": { "type": "string" }, "Encrypted": { "type": "boolean" }, "Iops": { "type": "number" }, "KmsKeyId": { "type": "string" }, "SnapshotId": { "type": "string" }, "Throughput": { "type": "number" }, "VolumeSize": { "type": "number" }, "VolumeType": { "type": "string" } }, "required": [ "DeviceName" ] }, "EnclaveOptions": { "type": "object", "additionalProperties": false, "properties": { "Enabled": { "type": "boolean" } }, "required": [] }, "EphemeralBlockDevice": { "type": "object", "additionalProperties": false, "properties": { "DeviceName": { "type": "string" }, "NoDevice": { "type": "boolean" }, "VirtualName": { "type": "string" } }, "required": [ "DeviceName" ] }, "MetadataOptions": { "type": "object", "additionalProperties": false, "properties": { "HttpEndpoint": { "type": "string" }, "HttpPutResponseHopLimit": { "type": "number" }, "HttpTokens": { "type": "string" } }, "required": [] }, "NetworkInterface": { "type": "object", "additionalProperties": false, "properties": { "DeleteOnTermination": { "type": "boolean" }, "DeviceIndex": { "type": "number" }, "NetworkInterfaceId": { "type": "string" } }, "required": [ "DeviceIndex", "NetworkInterfaceId" ] }, "RootBlockDevice": { "type": "object", "additionalProperties": false, "properties": { "DeleteOnTermination": { "type": "boolean" }, "Encrypted": { "type": "boolean" }, "Iops": { "type": "number" }, "KmsKeyId": { "type": "string" }, "Throughput": { "type": "number" }, "VolumeSize": { "type": "number" }, "VolumeType": { "type": "string" } }, "required": [] }, "Timeouts": { "type": "object", "additionalProperties": false, "properties": { "Create": { "type": "string" }, "Delete": { "type": "string" }, "Update": { "type": "string" } }, "required": [] } }, "properties": { "tfcfnid": { "description": "Internal identifier for tracking resource changes. Do not use.", "type": "string" }, "Ami": { "type": "string", "description": "The AMI to use for the instance." }, "Arn": { "type": "string" }, "AssociatePublicIpAddress": { "type": "boolean", "description": "Associate a public ip address with an instance in a VPC. Boolean value." }, "AvailabilityZone": { "type": "string", "description": "The AZ to start the instance in." }, "CpuCoreCount": { "type": "number", "description": "Sets the number of CPU cores for an instance. This option is\nonly supported on creation of instance type that support CPU Options\n[CPU Cores and Threads Per CPU Core Per Instance Type](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instance-optimize-cpu.html#cpu-options-supported-instances-values) - specifying this option for unsupported instance types will return an error from the EC2 API." }, "CpuThreadsPerCore": { "type": "number", "description": "If set to to 1, hyperthreading is disabled on the launched instance. Defaults to 2 if not set. See [Optimizing CPU Options](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instance-optimize-cpu.html) for more information." }, "DisableApiTermination": { "type": "boolean", "description": "If true, enables [EC2 Instance\nTermination Protection](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/terminating-instances.html#Using_ChangingDisableAPITermination)." }, "EbsOptimized": { "type": "boolean", "description": "If true, the launched EC2 instance will be EBS-optimized.\nNote that if this is not set on an instance type that is optimized by default then\nthis will show as disabled but if the instance type is optimized by default then\nthere is no need to set this and there is no effect to disabling it.\nSee the [EBS Optimized section](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EBSOptimized.html) of the AWS User Guide for more information." }, "GetPasswordData": { "type": "boolean", "description": "If true, wait for password data to become available and retrieve it. Useful for getting the administrator password for instances running Microsoft Windows. The password data is exported to the `password_data` attribute. See [GetPasswordData](https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_GetPasswordData.html) for more information." }, "Hibernation": { "type": "boolean", "description": "If true, the launched EC2 instance will support hibernation." }, "HostId": { "type": "string", "description": "The Id of a dedicated host that the instance will be assigned to. Use when an instance is to be launched on a specific dedicated host." }, "IamInstanceProfile": { "type": "string", "description": "The IAM Instance Profile to\nlaunch the instance with. Specified as the name of the Instance Profile. Ensure your credentials have the correct permission to assign the instance profile according to the [EC2 documentation](http://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html#roles-usingrole-ec2instance-permissions), notably `iam:PassRole`." }, "Id": { "type": "string" }, "InstanceInitiatedShutdownBehavior": { "type": "string", "description": "Shutdown behavior for the\ninstance. Amazon defaults this to `stop` for EBS-backed instances and\n`terminate` for instance-store instances. Cannot be set on instance-store\ninstances. See [Shutdown Behavior](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/terminating-instances.html#Using_ChangingInstanceInitiatedShutdownBehavior) for more information." }, "InstanceState": { "type": "string" }, "InstanceType": { "type": "string", "description": "The type of instance to start. Updates to this field will trigger a stop/start of the EC2 instance." }, "Ipv6AddressCount": { "type": "number", "description": "A number of IPv6 addresses to associate with the primary network interface. Amazon EC2 chooses the IPv6 addresses from the range of your subnet." }, "Ipv6Addresses": { "type": "array", "insertionOrder": false, "items": { "type": "string" }, "description": "Specify one or more IPv6 addresses from the range of the subnet to associate with the primary network interface." }, "KeyName": { "type": "string", "description": "The key name of the Key Pair to use for the instance; which can be managed using [the `aws_key_pair` resource](key_pair.html)." }, "Monitoring": { "type": "boolean", "description": "If true, the launched EC2 instance will have detailed monitoring enabled. (Available since v0.6.0)." }, "OutpostArn": { "type": "string" }, "PasswordData": { "type": "string" }, "PlacementGroup": { "type": "string", "description": "The Placement Group to start the instance in." }, "PrimaryNetworkInterfaceId": { "type": "string" }, "PrivateDns": { "type": "string" }, "PrivateIp": { "type": "string", "description": "Private IP address to associate with the\ninstance in a VPC." }, "PublicDns": { "type": "string" }, "PublicIp": { "type": "string" }, "SecondaryPrivateIps": { "type": "array", "insertionOrder": true, "items": { "type": "string" }, "description": "A list of secondary private IPv4 addresses to assign to the instance's primary network interface (eth0) in a VPC. Can only be assigned to the primary network interface (eth0) attached at instance creation, not a pre-existing network interface i.e. referenced in a `network_interface` block. Refer to the [Elastic network interfaces documentation](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-eni.html#AvailableIpPerENI) to see the maximum number of private IP addresses allowed per instance type." }, "SecurityGroups": { "type": "array", "insertionOrder": true, "items": { "type": "string" }, "description": "A list of security group names (EC2-Classic) or IDs (default VPC) to associate with." }, "SourceDestCheck": { "type": "boolean", "description": "Controls if traffic is routed to the instance when\nthe destination address does not match the instance. Used for NAT or VPNs. Defaults true." }, "SubnetId": { "type": "string", "description": "The VPC Subnet ID to launch in." }, "Tags": { "type": "array", "insertionOrder": true, "items": { "type": "object", "additionalProperties": false, "properties": { "MapKey": { "type": "string" }, "MapValue": { "type": "string" } }, "required": [ "MapKey", "MapValue" ] }, "description": "A map of tags to assign to the resource." }, "Tenancy": { "type": "string", "description": "The tenancy of the instance (if the instance is running in a VPC). An instance with a tenancy of dedicated runs on single-tenant hardware. The host tenancy is not supported for the import-instance command." }, "UserData": { "type": "string", "description": "The user data to provide when launching the instance. Do not pass gzip-compressed data via this argument; see `user_data_base64` instead." }, "UserDataBase64": { "type": "string", "description": "Can be used instead of `user_data` to pass base64-encoded binary data directly. Use this instead of `user_data` whenever the value is not a valid UTF-8 string. For example, gzip-encoded user data must be base64-encoded and passed via this argument to avoid corruption." }, "VolumeTags": { "type": "array", "insertionOrder": true, "items": { "type": "object", "additionalProperties": false, "properties": { "MapKey": { "type": "string" }, "MapValue": { "type": "string" } }, "required": [ "MapKey", "MapValue" ] }, "description": "A map of tags to assign to the devices created by the instance at launch time." }, "VpcSecurityGroupIds": { "type": "array", "insertionOrder": true, "items": { "type": "string" }, "description": "A list of security group IDs to associate with." }, "CreditSpecification": { "type": "array", "insertionOrder": false, "items": { "$ref": "#/definitions/CreditSpecification" }, "maxItems": 1 }, "EbsBlockDevice": { "type": "array", "insertionOrder": true, "items": { "$ref": "#/definitions/EbsBlockDevice" } }, "EnclaveOptions": { "type": "array", "insertionOrder": false, "items": { "$ref": "#/definitions/EnclaveOptions" }, "maxItems": 1 }, "EphemeralBlockDevice": { "type": "array", "insertionOrder": true, "items": { "$ref": "#/definitions/EphemeralBlockDevice" } }, "MetadataOptions": { "type": "array", "insertionOrder": false, "items": { "$ref": "#/definitions/MetadataOptions" }, "maxItems": 1 }, "NetworkInterface": { "type": "array", "insertionOrder": true, "items": { "$ref": "#/definitions/NetworkInterface" } }, "RootBlockDevice": { "type": "array", "insertionOrder": false, "items": { "$ref": "#/definitions/RootBlockDevice" }, "maxItems": 1 }, "Timeouts": { "$ref": "#/definitions/Timeouts" } }, "additionalProperties": false, "required": [ "Ami", "InstanceType" ], "readOnlyProperties": [ "/properties/tfcfnid", "/properties/Arn", "/properties/Id", "/properties/InstanceState", "/properties/OutpostArn", "/properties/PasswordData", "/properties/PrimaryNetworkInterfaceId", "/properties/PrivateDns", "/properties/PublicDns", "/properties/PublicIp" ], "primaryIdentifier": [ "/properties/tfcfnid" ], "handlers": { "create": { "permissions": [ "*" ] }, "read": { "permissions": [ "*" ] }, "update": { "permissions": [ "*" ] }, "delete": { "permissions": [ "*" ] }, "list": { "permissions": [ "*" ] } } }johnttompkins commentedon Jan 14, 2021
Unable to reproduce with a minimal stack with only tags and a primary identifier:
Schema:
Template for create:
template for update:
cloudformation-cli-python-lib-2.1.5
cloudformation-cli-python-plgin-2.1.2
cloudformation-cli 0.2.1
Handlers simply returned model input into the template back to cloudformation. it appears to be something with the handler code being submitted misinterpreting the tags on return. could you share the handler code for create/update?
sriram9707 commentedon Jan 15, 2021
Please find the handlers.py and models.py @jotompki
Handlers.py:-
`import logging
import json
import os
from uuid import uuid4
from typing import Any, MutableMapping, Optional
from cloudformation_cli_python_lib import (
Action,
HandlerErrorCode,
OperationStatus,
ProgressEvent,
Resource,
SessionProxy,
exceptions,
)
from .models import ResourceHandlerRequest, ResourceModel
LOG = logging.getLogger(name)
TYPE_NAME = "Terraform::AWS::Instance"
resource = Resource(TYPE_NAME, ResourceModel)
test_entrypoint = resource.test_entrypoint
def check_progress(operationid, trackingid, progress, session):
LOG.warn("Retrieving existing operation status ({})".format(operationid))
@resource.handler(Action.CREATE)
def create_handler(
session: Optional[SessionProxy],
request: ResourceHandlerRequest,
callback_context: MutableMapping[str, Any],
) -> ProgressEvent:
model = request.desiredResourceState
progress: ProgressEvent = ProgressEvent(
status=OperationStatus.IN_PROGRESS,
resourceModel=model,
)
@resource.handler(Action.UPDATE)
def update_handler(
session: Optional[SessionProxy],
request: ResourceHandlerRequest,
callback_context: MutableMapping[str, Any],
) -> ProgressEvent:
model = request.desiredResourceState
progress: ProgressEvent = ProgressEvent(
status=OperationStatus.IN_PROGRESS,
resourceModel=model,
)
@resource.handler(Action.DELETE)
def delete_handler(
session: Optional[SessionProxy],
request: ResourceHandlerRequest,
callback_context: MutableMapping[str, Any],
) -> ProgressEvent:
model = request.desiredResourceState
progress: ProgressEvent = ProgressEvent(
status=OperationStatus.IN_PROGRESS,
resourceModel=model,
)
@resource.handler(Action.READ)
def read_handler(
session: Optional[SessionProxy],
request: ResourceHandlerRequest,
callback_context: MutableMapping[str, Any],
) -> ProgressEvent:
model = request.desiredResourceState
@resource.handler(Action.LIST)
def list_handler(
session: Optional[SessionProxy],
request: ResourceHandlerRequest,
callback_context: MutableMapping[str, Any],
) -> ProgressEvent:
# TODO: put code here
return ProgressEvent(
status=OperationStatus.SUCCESS,
resourceModels=[],
)`
sriram9707 commentedon Jan 15, 2021
Models.py:-
`
import sys
from dataclasses import dataclass
from inspect import getmembers, isclass
from typing import (
AbstractSet,
Any,
Generic,
Mapping,
MutableMapping,
Optional,
Sequence,
Type,
TypeVar,
)
from cloudformation_cli_python_lib.interface import (
BaseModel,
BaseResourceHandlerRequest,
)
from cloudformation_cli_python_lib.recast import recast_object
from cloudformation_cli_python_lib.utils import deserialize_list
T = TypeVar("T")
def set_or_none(value: Optional[Sequence[T]]) -> Optional[AbstractSet[T]]:
if value:
return set(value)
return None
@DataClass
class ResourceHandlerRequest(BaseResourceHandlerRequest):
# pylint: disable=invalid-name
desiredResourceState: Optional["ResourceModel"]
previousResourceState: Optional["ResourceModel"]
@DataClass
class ResourceModel(BaseModel):
tfcfnid: Optional[str]
Ami: Optional[str]
Arn: Optional[str]
AssociatePublicIpAddress: Optional[bool]
AvailabilityZone: Optional[str]
CpuCoreCount: Optional[float]
CpuThreadsPerCore: Optional[float]
DisableApiTermination: Optional[bool]
EbsOptimized: Optional[bool]
GetPasswordData: Optional[bool]
Hibernation: Optional[bool]
HostId: Optional[str]
IamInstanceProfile: Optional[str]
Id: Optional[str]
InstanceInitiatedShutdownBehavior: Optional[str]
InstanceState: Optional[str]
InstanceType: Optional[str]
Ipv6AddressCount: Optional[float]
Ipv6Addresses: Optional[Sequence[str]]
KeyName: Optional[str]
Monitoring: Optional[bool]
OutpostArn: Optional[str]
PasswordData: Optional[str]
PlacementGroup: Optional[str]
PrimaryNetworkInterfaceId: Optional[str]
PrivateDns: Optional[str]
PrivateIp: Optional[str]
PublicDns: Optional[str]
PublicIp: Optional[str]
SecondaryPrivateIps: Optional[Sequence[str]]
SecurityGroups: Optional[Sequence[str]]
SourceDestCheck: Optional[bool]
SubnetId: Optional[str]
Tags: Optional[Sequence["_Tags"]]
Tenancy: Optional[str]
UserData: Optional[str]
UserDataBase64: Optional[str]
VolumeTags: Optional[Sequence["_VolumeTags"]]
VpcSecurityGroupIds: Optional[Sequence[str]]
CreditSpecification: Optional[Sequence["_CreditSpecification"]]
EbsBlockDevice: Optional[Sequence["_EbsBlockDevice"]]
EnclaveOptions: Optional[Sequence["_EnclaveOptions"]]
EphemeralBlockDevice: Optional[Sequence["_EphemeralBlockDevice"]]
MetadataOptions: Optional[Sequence["_MetadataOptions"]]
NetworkInterface: Optional[Sequence["_NetworkInterface"]]
RootBlockDevice: Optional[Sequence["_RootBlockDevice"]]
Timeouts: Optional["_Timeouts"]
_ResourceModel = ResourceModel
@DataClass
class Tags(BaseModel):
MapKey: Optional[str]
MapValue: Optional[str]
_Tags = Tags
@DataClass
class VolumeTags(BaseModel):
MapKey: Optional[str]
MapValue: Optional[str]
_VolumeTags = VolumeTags
@DataClass
class CreditSpecification(BaseModel):
CpuCredits: Optional[str]
_CreditSpecification = CreditSpecification
@DataClass
class EbsBlockDevice(BaseModel):
DeleteOnTermination: Optional[bool]
DeviceName: Optional[str]
Encrypted: Optional[bool]
Iops: Optional[float]
KmsKeyId: Optional[str]
SnapshotId: Optional[str]
Throughput: Optional[float]
VolumeSize: Optional[float]
VolumeType: Optional[str]
_EbsBlockDevice = EbsBlockDevice
@DataClass
class EnclaveOptions(BaseModel):
Enabled: Optional[bool]
_EnclaveOptions = EnclaveOptions
@DataClass
class EphemeralBlockDevice(BaseModel):
DeviceName: Optional[str]
NoDevice: Optional[bool]
VirtualName: Optional[str]
_EphemeralBlockDevice = EphemeralBlockDevice
@DataClass
class MetadataOptions(BaseModel):
HttpEndpoint: Optional[str]
HttpPutResponseHopLimit: Optional[float]
HttpTokens: Optional[str]
_MetadataOptions = MetadataOptions
@DataClass
class NetworkInterface(BaseModel):
DeleteOnTermination: Optional[bool]
DeviceIndex: Optional[float]
NetworkInterfaceId: Optional[str]
_NetworkInterface = NetworkInterface
@DataClass
class RootBlockDevice(BaseModel):
DeleteOnTermination: Optional[bool]
Encrypted: Optional[bool]
Iops: Optional[float]
KmsKeyId: Optional[str]
Throughput: Optional[float]
VolumeSize: Optional[float]
VolumeType: Optional[str]
_RootBlockDevice = RootBlockDevice
@DataClass
class Timeouts(BaseModel):
Create: Optional[str]
Delete: Optional[str]
Update: Optional[str]
_Timeouts = Timeouts
`
sriram9707 commentedon Feb 3, 2021
@jotompki is there any update please ?
johnttompkins commentedon Feb 4, 2021
Could you point to a github repo or something a bit more formatted? It is hard to tell how all of these are interacting in an issue comment.
sriram9707 commentedon Feb 5, 2021
@jotompki sure. please find the github repo https://github.com/iann0036/cfn-tf-custom-types.git
sriram9707 commentedon Mar 7, 2021
@jotompki is there any update on this pls ?
sriram9707 commentedon Mar 23, 2021
any update on this please ? @jotompki @ammokhov @wbingli @stilvoid
JohnPreston commentedon Feb 10, 2022
I am having similar issues with an array of simple objects
I have it down to an
IndexError. However, all SAM local tests andaws cloudformation test-typework just fine.It only is when using "the real CFN" that it fails with the exact same input (i.e, took what's in inputs/input_1_create.json and set that as
Propertiesin the template to try it with).note: the IndexError is caught by
Exceptioninresources.py.Resource.__call__I had to modify that file to have the ProgressEvent return the message